:W&Bá¾êK4ÍM wclzbk8uª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0aO\¨ƒ‚" zyxe5mwkocqq wclzbk8uŠß wclzbk8uescheuller-uc-san-diego spatop-sweep"ž _wandb‚{}  batch_size‚2048  dropout‚0.24857038731466477  hidden_dim‚128 ' learning_rate‚0.0004490769426577584  num_embedding_layers‚6Bdpfx_vanilla_wclzbk8ubnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ ô¾ëÒ «Ñ±š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64 K“¯; wclzbk8u*  _runtime‚0  _wandb‚ {"runtime":0}/ÀƒbÒÜ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 ô¾ëÒ€ñ¨î*-of*>/data/spatop/options_files/sweep_fixed_dp/config_wclzbk8u.json*-l* /scratch/logs*-n*dpfx_vanilla_wclzbk8u2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº sq100y3h8yit14ohvyimypn1dybugj9pHÅ+éÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€­Þª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º sq100y3h8yit14ohvyimypn1dybugj9pµª½Ç2  requirements.txtKŠü»W wclzbk8uZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_640ÑÀY wclzbk8uZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64Éõ—¦*  wclzbk8ub trainer/global_step2fÕÐ\  wclzbk8uZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64Û'zç/  wclzbk8ub*"trainer/global_step2E_õK  wclzbk8uj: õ¾ëҨɌÌ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ËmÓk“  wclzbk8uj  ÷¾ëÒˆò–—î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ Àá¿/\ wclzbk8ujÊ ÷¾ëÒ ÈȘ·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.814 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 Òr$žq wclzbk8uj` ÷¾ëÒØÆú˜Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). pæ_Ý{ wclzbk8ujj ÷¾ëÒ¨¦Š™Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). FÏ,;q wclzbk8uj` ÷¾ëÒ€³˜™Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). e¼qÂt wclzbk8ujc ÷¾ëÒݤ™Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ˜`‡®n wclzbk8uj] ÷¾ëҀ谙Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ©e®µð wclzbk8u*Þ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"à»ú`ë wclzbk8ujÙ ÷¾ëÒÐׂžÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. _z³9N wclzbk8uj¼ ÷¾ëÒàΟž«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. À@¨ë wclzbk8ujÙ ü¾ëÒÀ¥úâÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. IêFÉP wclzbk8uj¾ ü¾ëÒ Êã­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ؼŠÖ“‚(:‰ „¿ëÒ¦æï“ gpu.0.gpu‚100.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.777984619140625* gpu.0.memoryAllocatedBytes‚ 25579094016 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Ά›‚(:‘ „¿ëÒÑƸš) disk./.usagePercent‚40.23509182109588 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 6.7822265625 disk.nvme0c0n1.in‚ 0.09765625 network.recv‚41720 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 0.4453125 disk.dm-0.in‚ 0.09765625 disk.sda3.out‚ 0.4453125) proc.memory.percent‚16.06919765472412 disk.rbd1.out‚0.0625 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 6.7822265625 disk.dm-0.out‚ 6.7822265625 disk.sda.out‚ 0.4453125 network.sent‚33335 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 0.09765625 disk.rbd1.in‚0# proc.memory.availableMB‚4.78125# proc.memory.rssMB‚ 2632.77734375 cpu‚25.780615700112186 disk.drbd2165.out‚0$ memory_percent‚99.97081756591797$ disk./.usageGB‚599.5319442749023 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17´ž‚(:” “¿ëÒ‰Ú™“ gpu.0.gpu‚60.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ,Ç=—‚(: “¿ëҟʘ• proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 0.8046875 cpu‚25.90956383744139 network.sent‚43896 disk.dm-0.out‚ 32.5400390625* proc.memory.percent‚16.069459915161133 disk.sda3.out‚ 0.8046875 disk.rbd1.out‚0.0625( proc.memory.availableMB‚ 3484.8671875* disk./.usagePercent‚40.121107026550725 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0" disk.nvme0n1.out‚ 32.5400390625 disk.rbd1.in‚0$ disk./.usageGB‚597.8334884643555 disk.sda2.out‚0 disk.nvme0n1.in‚1.0625 disk.sda.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 32.5400390625 network.recv‚45957 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚1.0625 disk.sda.out‚ 0.8046875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚1.0625$ memory_percent‚78.73005867004395" proc.memory.rssMB‚ 2632.8203125Ï)×4“‚(:‰ ¢¿ëÒ¶ Ü” gpu.0.gpu‚84.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ω~C£‚(:™ ¢¿ëÒ…š‡– proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 1.89453125 disk.dm-0.in‚ 1.89453125 disk.nvme0n1.in‚ 1.89453125 disk.sda1.in‚0 disk.dm-0.out‚ 85.455078125* proc.memory.percent‚16.069531440734863 network.sent‚47456) disk./.usagePercent‚40.02675468178386 disk.sda.out‚ 1.41015625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1.41015625$ memory_percent‚78.73225212097168( proc.memory.availableMB‚ 3484.5078125 disk.rbd1.out‚0.1875 disk.sda3.out‚ 1.41015625# proc.memory.rssMB‚ 2632.83203125 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.99511937744591 network.recv‚49673$ disk./.usageGB‚596.4275703430176! disk.nvme0n1.out‚ 85.455078125 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 85.455078125ˆCy3b/ lr-AdamW/pg1"trainer/global_step2:JPnýQß3 b/ lr-AdamW/pg2"trainer/global_step2:JP„A[\-!) _wandbruntime‚51  _runtime‚51ÎrŸ—Â"½ % _timestamp‚1.7843403923857927e+09 _runtime‚ 51.079480482 _step‚0 ' lr-AdamW/pg1‚5.5011925475575406e-05 ' lr-AdamW/pg2‚5.5011925475575406e-05 trainer/global_step‚49|¯Ó‰Ä#¿ _runtime‚ 51.079480482 _step‚0 ' lr-AdamW/pg1‚5.5011925475575406e-05 ' lr-AdamW/pg2‚5.5011925475575406e-05 trainer/global_step‚49 % _timestamp‚1.7843403923857927e+09%‰Ýp$bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPUéß@l%bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPÃvs|g&bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP¨þУt'bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPCÝ^(bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP/W)bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPš™óf*bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPÆ*¨^+bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPhÊB­f,bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP¶;&f-bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPú]¬ˆo.bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP~0%h/bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPíAÝi0be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP¡zÂc1b_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPÇ‚·Àkbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPvê‘Sp?bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPpš‚ll@bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP‡±Z(hAbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPÒzÃÎfBbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPeA¤nCbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP‡ð%]sDbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPàýh7iEbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP#¢UnFbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP*v_Gb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP+tY|pHbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP½lq±qIbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPB°«pJbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPªPÓcKb_ 5j\bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP}50~n]bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPªÍZ‘p^bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP””¿q_bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPãb+e`ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP ºÉ‹nabj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP4&Œmbbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPèD©n_cb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPR¨¬£sdbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP™X_eb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP§›kcfb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPH •kybg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPU BÇizbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP-c¿Jm{bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPWVx|bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPiwâj}bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÓѤža~b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP¨7*²kbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP—Àéßs€bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP&ÒÎUsbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP$‡'ˆj‚be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPlêy‡lƒbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPk^i„bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP~ìAïh…bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPÊíÞ¼m†bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPß4k‡bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPzþhjˆbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPD©Qðr‰bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP´ì»œhŠbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP©”.m‹bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPTÎHmŒbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP2¸L«obj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPŠ@4ÄaŽb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPÊ6âhbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPH¥Ÿkbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP‡ßAÝp‘bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPÅï~œo’bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPɳü”^“bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP?YT(k–bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPŠmqKf—ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPû2çöm˜bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPEÇáÅo™bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP>I³ƒhšbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPÄÖy›bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPxû»cjœbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPbúÝgbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPwöÌ€gžbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPh×kQ.Ÿ) _wandbruntime‚51  _runtime‚51ÿsiGÖ €O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1795416803328 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 trainer/global_step‚49 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 51207128668 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055055 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7397875425280 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚461154 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚202026 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 52661682176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 % _timestamp‚1.7843403923870015e+09 _step‚1 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7449520251904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702768640 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 52587605504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11801 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360800370688 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7449520251904 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚202287 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 51286627840 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7412007499356 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚96795136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚836988 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 QGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7397892464640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261ùÎ(6Š'¡„O \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51627787264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7449162053120 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚ 51.080846741 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7397892464640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 QGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7397892464640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1795416803328 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 trainer/global_step‚49 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 51207128668 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055055 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7397875425280 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚461154 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚202026 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 52661682176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚513 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 % _timestamp‚1.7843403923870015e+09 _step‚1 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7449520251904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702768640 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 52587605504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11801 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360800370688 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7449520251904 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚202287 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 51286627840 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7412007499356 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚96795136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚836988 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 GDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPìuqi§bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP$9l¨bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP­Ç-b©b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPùþR{eªb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPì÷Àq«bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPCkr¬bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP«Êb­b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP¾þæ]®bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPBð™+e¯b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP‘íÂËm°bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPFbl±bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP[¾nSi²bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPzàýÅf³ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPþT¬F^´bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP Žùhµbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP¬^Q‡^¶bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP»"TÞs·bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP`æµf¸ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÙú m¹bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPƒmrkºbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPBcÒ7]»bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP6G˜Òk¼bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP-¼lg½bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPêíŒoe¾b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPQgÌþo¿bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP%ëÏjÀbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPŸQ<0jÁbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPÖ ÿmÂbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPTä§×rÃbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP ,nÔpÄbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÔÔ¤TiÅbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPä„ ÝgÆbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPñ'G›eÇb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPF1 wdÈb_ ·odÉb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP*&Jkîbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPÍFU´eïb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPn!ilðbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPA©Ýhñbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPз8ópòbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPEºÑÔmóbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPò´uôbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPôèè‰_õbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP½è£;göbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP`Ðmo÷bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPaQk‡nøbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPð?ô1dùb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP7mnÛq€bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP9¿Iæmbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPT„²l‚bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPŒ‰Ifƒba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP“ÀÏú`„b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP™ÉÎõc…b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPáGŽf†ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPŸÌAgu‡bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPõ^’iˆbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPH hUh‰bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPoBߢiŠbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP5’Kã`‹b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPmWú2mŒbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP*@7hbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPªœ²»kŽbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÁܧqbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPY¯jVbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPu„ë§e‘b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP¹Îx2m’bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP Hî/h“bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP©±ún”bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÉ»î3p•bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP'ƒCü\–bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPå G‚o—bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPe™†Wc˜b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP ò]‡h™bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPÄIUöfšba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPW Ñ.r›bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPþnúrœbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPÉN³.) _wandbruntime‚51  _runtime‚51HXyhž‹M PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚470220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚2 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1828826949632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7559529224950 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 52299254784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.alc5V)l.freed‚ 1882441566208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚460580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚111189 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1882542636544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 trainer/global_step‚49 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393094 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚96795136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 52218182390 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0º©8•&ŸM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 OEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94873 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 52299254784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚460580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚111189 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1882542636544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 trainer/global_step‚49 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393094 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚96795136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 52218182390 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚470220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚2 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1828826949632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7559529224950 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 B loss/SRqqt1/assignment_loss"trainer/global_step2:JPê-jC¨b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPòøåB©b= loss/SRqqt1/detection_loss"trainer/global_step2:JP6ðÌ6Aªb< loss/FRt1/assignment_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JPñy@¬b; loss/FRt1/detection_loss"trainer/global_step2:JPYd1@­b; loss/FRt2/detection_loss"trainer/global_step2:JPPRÐÆ.®) _wandbruntime‚51  _runtime‚51uñëŸõ¯ï 0loss/SRqqt2/detection_loss‚1.513945629996634 /loss/FBt1/detection_loss‚25.144289204882362 .loss/FBt2/detection_loss‚10.01944025679733 epoch‚0 trainer/global_step‚49 $ _timestamp‚1.784340393004386e+09 /loss/FRt2/assignment_loss‚5.331666757463611 1loss/SRqqt1/assignment_loss‚4.222250275611877 1loss/SRqqt2/assignment_loss‚4.209080726987472 0loss/SRqqt1/detection_loss‚4.645825145443281 /loss/FRt1/assignment_loss‚4.490796140408448 %loss/total_loss‚65.25163643262182 _runtime‚ 51.69767275 _step‚3 /loss/FRt1/detection_loss‚0.4171144842357144 .loss/FRt2/detection_loss‚1.575355523247197 0loss/FBt1/assignment_loss‚1.8452632851519828 0loss/FBt2/assignment_loss‚1.8366090023959125â%÷Åù°ó 0loss/FBt1/assignment_loss‚1.8452632851519828 0loss/FBt2/assignment_loss‚1.8366090023959125 0loss/SRqqt2/detection_loss‚1.513945629996634 /loss/FBt1/detection_loss‚25.144289204882362 .loss/FBt2/detection_loss‚10.01944025679733 epoch‚0 trainer/global_step‚49 $ _timestamp‚1.784340393004386e+09 /loss/FRt2/assignment_loss‚5.331666757463611 1loss/SRqqt1/assignment_loss‚4.222250275611877 1loss/SRqqt2/assignment_loss‚4.209080726987472 0loss/SRqqt1/detection_loss‚4.645825145443281 /loss/FRt1/assignment_loss‚4.490796140408448 %loss/total_loss‚65.25163643262182 _runtime‚ 51.69767275 _step‚3 /loss/FRt1/detection_loss‚0.4171144842357144 .loss/FRt2/detection_loss‚1.575355523247197é±”±‚(:‰ ±¿ëÒÑÌÀ“ gpu.0.gpu‚82.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[kH•²‚(:Š ±¿ëÒñåä—& proc.memory.availableMB‚ 3482.59375 disk.drbd2165.in‚0! disk.nvme0c0n1.out‚ 88.6328125 disk.rbd1.out‚0.1875 disk.sda1.in‚0 network.sent‚58967* disk./.usagePercent‚40.026765434100305$ disk./.usageGB‚596.4277305603027 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-0.in‚108.3125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1.7578125 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚108.3125 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.069912910461426 network.recv‚55991 disk.sda.out‚ 1.7578125# proc.memory.rssMB‚ 2632.89453125 cpu‚26.068124647667887 disk.dm-0.out‚ 88.6328125 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚78.74393463134766 disk.nvme0n1.in‚108.3125 disk.nvme0n1.out‚ 88.6328125 disk.sda3.out‚ 1.7578125¶h锳‚(:‰ À¿ëÒɘÓ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¤ü(˜´‚(: À¿ëÒ„÷®– disk.sda1.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2632.90625 disk.sda.out‚2.375 disk.nvme0n1.in‚ 109.27734375 disk.drbd2165.in‚0$ disk.nvme0c0n1.out‚ 99.7646484375 disk.dm-0.in‚ 109.27734375 cpu‚26.107216051971054 network.sent‚106380) disk./.usagePercent‚40.02677183428866 disk.dm-1.out‚2.375 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 99.7646484375 disk.sda.in‚0 disk.rbd1.out‚0.25% proc.memory.availableMB‚ 3481.3125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 109.27734375" disk.nvme0n1.out‚ 99.7646484375 network.recv‚60592$ disk./.usageGB‚596.4278259277344 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.069984436035156$ memory_percent‚78.75175476074219 disk.dm-1.in‚0 disk.sda3.out‚2.375 disk.rbd0.in‚0ñT$‘µ‚(:† Ï¿ëÒç©Ï“ gpu.0.gpu‚26.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0| ’¶‚(:‡ Ï¿ëÒýŠ–% disk.nvme0c0n1.out‚107.8017578125 disk.sda.in‚0 disk.sda.out‚2.5 disk.nvme0n1.in‚ 110.109375 disk.sda3.out‚2.5* disk./.usagePercent‚40.026760313949616 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 110.109375 proc.memory.rssMB‚ 2632.90625* proc.memory.percent‚16.069984436035156 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 110.109375 disk.dm-0.out‚107.8017578125 disk.rbd1.in‚0 network.sent‚110916 disk.dm-1.out‚2.5 disk.sda2.in‚0$ memory_percent‚78.74999046325684 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚64013 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.out‚0.25 cpu‚26.13801709706546# disk.nvme0n1.out‚107.8017578125( proc.memory.availableMB‚ 3481.6015625$ disk./.usageGB‚596.4276542663574 disk.sda1.out‚0ž× .·) _wandbruntime‚91  _runtime‚91ôÒ²ÎÀ¸º trainer/global_step‚99 $ _timestamp‚1.784340432790847e+09 _runtime‚ 91.484319853 _step‚4 & lr-AdamW/pg1‚0.0001111465433077952 & lr-AdamW/pg2‚0.0001111465433077952~7}yĹ¾ $ _timestamp‚1.784340432790847e+09 _runtime‚ 91.484319853 _step‚4 & lr-AdamW/pg1‚0.0001111465433077952 & lr-AdamW/pg2‚0.0001111465433077952 trainer/global_step‚99†«P.º) _wandbruntime‚91  _runtime‚91¤cÝ¢'»œO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚915606 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3570861648896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 trainer/global_step‚99 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14788515167586 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14862267235328 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843404327916405e+09 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚394121 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 104114288128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÛA‘-á¼ O _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 XEDeviceStatsMoZ¶-Å#nitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚915606 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3570861648896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 trainer/global_step‚99 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14788515167586 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14862267235328 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843404327916405e+09 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚394121 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1041142881288\$ÿ.½) _wandbruntime‚92  _runtime‚92Q¾Åã¬&¾¦M UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 % _timestamp‚1.7843404334059155e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14907184975872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3500264876032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚99 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 _runtime‚ 92.099665504 _step‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774694 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692Ó´~h°&¿ªM J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚99 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 _runtime‚ 92.099665504 _step‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774694 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 % _timestamp‚1.7843404334059155e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14907184975872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3500264876032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268Êæø.À) _wandbruntime‚92  _runtime‚92\7 JúÁô 1loss/SRqqt1/assignment_loss‚2.887989578733019 /loss/FRt2/detection_loss‚1.5675492033051948 %loss/total_loss‚61.77379349105654 epoch‚0 % _timestamp‚1.7843404334090106e+09 0loss/FRt2/assignment_loss‚5.3680922329425815 1loss/FBt2/assignment_loss‚0.23035889692703046 .loss/FBt1/detection_loss‚31.30979610412665 trainer/global_step‚99 _step‚7 /loss/FBt1/assignment_loss‚0.429598237256236 /loss/FRt1/detection_loss‚0.3940321642179107 .loss/FBt2/detection_loss‚6.922283431646899 _runtime‚ 92.102299104 /loss/FRt1/assignment_loss‚4.512552197687632 1loss/SRqqt2/assignment_loss‚2.862860554786507 1loss/SRqqt1/detection_loss‚4.2660588530503265 1loss/SRqqt2/detection_loss‚1.0226220363765508‡"•æþÂø epoch‚0 % _timestamp‚1.7843404334090106e+09 0loss/FRt2/assignment_loss‚5.3680922329425815 1loss/FBt2/assignment_loss‚0.23035889692703046 .loss/FBt1/detection_loss‚31.30979610412665 trainer/global_step‚99 _step‚7 /loss/FBt1/assignment_loss‚0.429598237256236 /loss/FRt1/detection_loss‚0.3940321642179107 .loss/FBt2/detection_loss‚6.922283431646899 _runtime‚ 92.102299104 /loss/FRt1/assignment_loss‚4.512552197687632 1loss/SRqqt2/assignment_loss‚2.862860554786507 1loss/SRqqt1/detection_loss‚4.2660588530503265 1loss/SRqqt2/detection_loss‚1.0226220363765508 1loss/SRqqt1/assignment_loss‚2.887989578733019 /loss/FRt2/detection_loss‚1.5675492033051948 %loss/total_loss‚61.77379349105654XŸL+ŸÃ‚(:” Þ¿ëÒí½°“ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.925' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.785000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ɖ‘Š©Ä‚(:ž Þ¿ëÒôñ‹— network.sent‚116328$ disk.nvme0c0n1.out‚ 113.517578125 disk.sda.out‚ 2.88671875# memory_percent‚78.7517786026001# proc.memory.rssMB‚ 2632.91015625 proc.cpu.threads‚17 disk.rbd1.in‚0) proc.memory.availableMB‚ 3481.30859375$ disk./.usageGB‚596.4278564453125 disk.sda1.out‚0 disk.dm-1.out‚ 2.88671875" disk.nvme0n1.out‚ 113.517578125 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 110.109375 disk.dm-0.out‚ 113.517578125 network.recv‚67521 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚16.070008277893066 disk.sda.in‚0 cpu‚26.16325318042939 disk.dm-0.in‚ 110.109375 disk.sda2.out‚0 disk.sda3.out‚ 2.88671875* disk./.usagePercent‚40.026773882348934 disk.rbd1.out‚0.3125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 110.109375¥tx”Å‚(:‰ í¿ëÒ„ëø” gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚248.333' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷™šbÆ‚(:§ í¿ëÒÚ‡‰— disk.sda3.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.070032119750977 network.sent‚163837 disk.drbd2165.out‚0 disk.dm-0.out‚120.8330078125 disk.sda.in‚0# disk.nvme0n1.out‚120.8330078125" proc.memory.rssMB‚ 2632.9140625* disk./.usagePercent‚40.027094403781796 disk.rbd0.out‚0 Í(”Pdisk.sda3.out‚ 3.06640625 disk.rbd0.in‚0$ memory_percent‚78.75292301177979 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 199.95703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚0.3125% disk.nvme0c0n1.out‚120.8330078125 disk.dm-0.in‚ 199.95703125 disk.sda.out‚ 3.06640625 network.recv‚72632$ disk./.usageGB‚596.4326324462891 disk.dm-1.out‚ 3.06640625 disk.sda2.in‚0 disk.nvme0n1.in‚ 199.95703125 cpu‚26.19049610656361 proc.cpu.threads‚17 disk.sda2.out‚0) proc.memory.availableMB‚ 3481.12109375º`Ù0Ç+ _wandbruntime‚131  _runtime‚131Ô¼'ÅÈ¿ ' lr-AdamW/pg1‚0.00016728116114001501 ' lr-AdamW/pg2‚0.00016728116114001501 trainer/global_step‚149 % _timestamp‚1.7843404731247718e+09 _runtime‚ 131.818339001 _step‚8 p)qÉÉà _runtime‚ 131.818339001 _step‚8 ' lr-AdamW/pg1‚0.00016728116114001501 ' lr-AdamW/pg2‚0.00016728116114001501 trainer/global_step‚149 % _timestamp‚1.7843404731247718e+09 d¦30Ê+ _wandbruntime‚131  _runtime‚131é&«'Ë¥O WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 _runtime‚ 131.81959974 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 152677631488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5138292656128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5293733955072 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 155459911168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22011884597248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22121978388480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843404731258552e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1360432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚267631 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚267628 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22121995427840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90853888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 152780650846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22164665248094 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚9 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 trainer/global_step‚149 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5293843421184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1360432 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 153018790912 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22275014218752 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968øÅ!¯'Ì©O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22275014218752 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 _runtime‚ 131.81959974 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 152677631488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5138292656128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5293733955072 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 155459911168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22011884597248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22121978388480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843404731258552e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1360432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚267631 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚267628 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22121995427840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90853888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 152780650846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22164665248094 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚9 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 trainer/global_step‚149 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5293843421184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1360432 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 153018790912 É…çÌ0Í+ _wandbruntime‚132  _runtime‚132›A°–¶&ΰM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 154031425024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1369484 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_p!ω‹~!ool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 154031425024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1369484 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 E«Ø‚(:  šÀëÒÁ‰¾–$ disk.nvme0n1.out‚1312.4072265625# proc.memory.rssMB‚ 2632.92578125 proc.cpu.threads‚17 disk.dm-1.out‚ 4.0859375 disk.sda.out‚ 4.0859375 disk.nvme0n1.in‚ 201.765625) proc.memory.availableMB‚ 2509.19140625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚1312.4072265625 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 201.765625 disk.sda3.out‚ 4.0859375 network.sent‚222555 disk.sda3.in‚0# memory_percent‚84.6851110458374 cpu‚26.227704285120375) disk./.usagePercent‚40.10318675516144# disk./.usageGB‚597.566463470459 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.dm-0.out‚1312.4072265625 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.recv‚85817 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.4375* proc.memory.percent‚16.070103645324707 disk.sda1.in‚0 disk.dm-0.in‚ 201.765625T)Ö¹0Ù+ _wandbruntime‚172  _runtime‚172o×8ÃÚ½ & lr-AdamW/pg1‚0.0002234157789722348 & lr-AdamW/pg2‚0.0002234157789722348 trainer/global_step‚199 $ _timestamp‚1.784340513846759e+09 _runtime‚ 172.540325435 _step‚12ÁÀ­†ÇÛÁ trainer/global_step‚199 $ _timestamp‚1.784340513846759e+09 _runtime‚ 172.540325435 _step‚12 & lr-AdamW/pg1‚0.0002234157789722348 & lr-AdamW/pg2‚0.0002234157789722348 ä :®0Ü+ _wandbruntime‚172  _runtime‚172ÙZÉ(­'ݧO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29484046909440 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 206904713728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29540815328602 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 206904713728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29540815328602 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 HDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29631270899712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204385760256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204726926848 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6843159341056 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚14 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 trainer/global_step‚199 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204385760256 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1833576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29687979466474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29688337054308 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6843140728832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824Æ;»&áµM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6843140728832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29835656659968 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1823936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚782684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 % _timestamp‚1.7843405144631817e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29836014865920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29631270899712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204385760256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204726926848 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6843159341056 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚14 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 trainer/global_step‚199 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204385760256 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1833576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29687979466474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29688337054308 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 (ü<0â+ _wandbruntime‚173  _runtime‚173î–ôþãø .loss/FRt2/assignment_loss‚4.61414860709896 2loss/SRqqt2/assignment_loss‚2.1503626985656057 .loss/FRt2/detection_loss‚1.450763778068216 &loss/total_loss‚44.410010778457355 1loss/FBt1/assignment_loss‚0.25499687382967573 1loss/FBt2/assignment_loss‚0.12512098180727038 /loss/FRt1/detection_loss‚0.3614787954541379 1loss/SRqqt2/detection_loss‚1.1105995971564502 .loss/FBt2/detection_loss‚5.326630837885702 epoch‚0 % _timestamp‚1.7843405144656236e+09 /loss/FRt1/assignment_loss‚4.549843045058968 1loss/SRqqt1/assignment_loss‚2.374048463921798 .loss/FBt1/detection_loss‚19.04029496736265 _runtime‚ 173.158927825 _step‚15 1loss/SRqqt1/detection_loss‚3.0517221322479218 trainer/global_step‚199¤åÄ¿äü 2loss/SRqqt2/assignment_loss‚2.1503626985656057 .loss/FRt2/detection_loss‚1.450763778068216 &loss/total_loss‚44.410010778457355 1loss/FBt1/assignment_loss‚0.25499687382967573 1loss/FBt2/assignment_loss‚0.12512098180727038 /loss/FRt1/detection_loss‚0.3614787954541379 1loss/SRqqt2/detection_loss‚1.1105995971564502 .loss/FBt2/detection_loss‚5.326630837885702 epoch‚0 % _timestamp‚1.7843405144656236e+09 /loss/FRt1/assignment_loss‚4.549843045058968 1loss/SRqqt1/assignment_loss‚2.374048463921798 .loss/FBt1/detection_loss‚19.04029496736265 _runtime‚ 173.158927825 _step‚15 1loss/SRqqt1/detection_loss‚3.0517221322479218 trainer/global_step‚199 .loss/FRt2/assignment_loss‚4.61414860709896uo”’å‚(:‡ ©ÀëÒ§¥ä“ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 šjâ³æ‚(:¨ ©ÀëÒÍÛ¿• disk.drbd2165.out‚0 disk.sda3.out‚ 4.37890625 disk.dm-0.out‚1316.189453125) proc.memory.availableMB‚ 2507.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚40.100420849761505 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚597.5252494812012 disk.nvme0n1.in‚ 306.86328125 disk.rbd1.in‚0 disk.dm-1.out‚ 4.37890625% disk.nvme0c0n1.out‚1316.189453125 disk.dm-0.in‚ 306.86328125$ memory_percent‚84.69316959381104# proc.memory.rssMB‚ 2632.92578125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 306.86328125* proc.memory.percent‚16.070103645324707 cpu‚26.24034471170342 proc.cpu.threads‚17 disk.sda.out‚ 4.37890625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 network.sent‚228012# disk.nvme0n1.out‚1316.189453125 network.recv‚89883 disk.rbd1.out‚0.4375 disk.sda1.out‚0®áŸç‚(:” ¸ÀëÒ«Üä“ gpu.0.gpu‚87.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚241.365' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.272999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝŒÌY«è‚(:  ¸ÀëÒÙ±•– disk.sda3.out‚4.765625 disk.rbd1.in‚0 disk.dm-0.in‚ 307.6953125" proc.memory.rssMB‚ 2632.9296875 disk.nvme0n1.in‚ 307.6953125% disk.nvme0n1.out‚1327.42626953125! disk.nvme0c0n1.in‚ 307.6953125" disk.dm-0.out‚1327.42626953125$ proc.memory.availableMB‚2507.625 network.sent‚275559 network.recv‚94671 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚84.69467163085938 cpu‚26.243062665893095 disk.sda.out‚4.765625* disk./.usagePercent‚40.031813134652914 disk.sda1.in‚0 disk.dm-1.out‚4.765625' disk.nvme0c0n1.out‚1327.42626953125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.070127487182617$ disk./.usageGB‚596.5029449462891 disk.rbd1.out‚0.5625 disk.rbd0.out‚0 disk.drbd2165.out‚0ÎÑ+Äé‚(:’ ÇÀëÒŒ£—• gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚190.712' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.142399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q³“­ê‚(:¢ ÇÀëÒá­•– disk.sda1.in‚0# disk.nvme0c0n1.out‚ 1336.8359375* proc.memory.percent‚16.070151329040527" disk.nvme0c0n1.in‚ 308.66015625# proc.memory.rssMB‚ 2632.93359375) proc.memory.availableMB‚ 2509.87890625 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚84.68091487884521 disk.sda3.in‚0 disk.sda3.out‚ 4.98046875 disk.rbd0.out‚0 disk.dm-0.in‚ 308.66015625) disk./.usagePercent‚40.02918828940435 disk.nvme0n1.in‚ 308.66015625 disk.sda.out‚ 4.98046875 disk.sda2.in‚0 cpu‚26.253969693779926 disk.rbd1.out‚0.5625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 network.sent‚281153 network.recv‚98633$ disk./.usageGB‚596.4638328552246 disk.dm-0.out‚ 1336.8359375 proc.cpu.threads‚17! disk.nvme0n1.out‚ 1336.8359375 disk.rbd1.in‚0 disk.dm-1.out‚ 4.98046875 disk.sda.in‚0HÂk£0ë+ _wandbruntime‚212  _runtime‚212ôG:´Åì¿ ' lr-AdamW/pg1‚0.00027955039680445463 ' lr-AdamW/pg2‚0.00027955039680445463 trainer/global_step‚249 % _timestamp‚1.7843405542787833e+09 _runtime‚ 212.97233119 _step‚16lwmfÉíà ' lr-AdamW/pg1‚0.00027955039680445463 ' lr-AdamW/pg2‚0.00027955039680445463 trainer/global_step‚249 % _timestamp‚1.7843405542787833e+09 _runtime‚ 212.97233119 _step‚16ø⇴0î+ _wandbruntime‚212  _runtime‚212²¦œLàï§O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2269336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2278962 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1911860 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚442131 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95048192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1911862 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚970044 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36916965409110 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37100149986816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36662951784448 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4190824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257974641664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚970292 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚442128 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1911862 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37100508185600 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 254068635136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37100508185600 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 254068635136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36846098391040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36916607821388 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911860 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all¨Î&Í.current‚9628 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 212.976435767 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8481149896704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36846081351680 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 254409794560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843405542795732e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8481168508928 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257879593472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911860 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 212.976435767 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8481149896704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36846081351680 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 254409794560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843405542795732e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8481168508928 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257879593472 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784340554894248e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 _step‚18 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚ 213.58803527 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36993322381312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 255422428672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37248761849344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2288028 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784340554894248e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 _step‚18 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚ 213.58803527 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36993322381312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 255422428672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37248761849344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I>Devicêì+? ceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2288028 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304630111570 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 _runtime‚ 253.427632199 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44513255169024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843405947338996e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115757568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2J·b·'€±O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2733414 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43988485378048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304630111570 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 _runtime‚ 253.427632199 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44513255169024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843405947338996e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115757568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚529378 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268¾n™k0+ _wandbruntime‚254  _runtime‚2542n ‚·M FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44440637215324 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843405953493328e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 305776763904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitoOœä/Ÿr.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 310159502848 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 _step‚22 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 254.043041863 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2301092 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10496194696704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 trainer/global_step‚299 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44661508832768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44661150626816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2742480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 305776763904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44134979010560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚634569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824r”+fÁ&ƒ»M EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44661150626816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2742480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 305776763904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44134979010560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚634569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44440637215324 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843405953493328e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 305776763904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 310159502848 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 _step‚22 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 254.043041863 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2301092 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10496194696704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 trainer/global_step‚299 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44661508832768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Nu|0„+ _wandbruntime‚254  _runtime‚254Òÿ…ú 1loss/FBt1/assignment_loss‚0.18827747634847813 .loss/FRt2/detection_loss‚1.441835538436372 .loss/FBt1/detection_loss‚5.786638126119547 epoch‚0 2loss/SRqqt1/assignment_loss‚1.6537141222402703 0loss/FBt2/assignment_loss‚0.1548958984632169 0loss/FRt1/detection_loss‚0.44030244179842754 1loss/SRqqt1/detection_loss‚2.8085765837118264 trainer/global_step‚299 /loss/FRt1/assignment_loss‚4.063876603554035 &loss/total_loss‚30.645059202661024 _runtime‚ 254.045601566 0loss/SRqqt2/detection_loss‚1.090827266438405 .loss/FBt2/detection_loss‚6.120491687881322 % _timestamp‚1.7843405953519766e+09 _step‚23 0loss/FRt2/assignment_loss‚5.2855938806962435 2loss/SRqqt2/assignment_loss‚1.6100295769728823ð‘Þ†þ epoch‚0 2loss/SRqqt1/assignment_loss‚1.6537141222402703 0loss/FBt2/assignment_loss‚0.1548958984632169 0loss/FRt1/detection_loss‚0.44030244179842754 1loss/SRqqt1/detection_loss‚2.8085765837118264 trainer/global_step‚299 /loss/FRt1/assignment_loss‚4.063876603554035 &loss/total_loss‚30.645059202661024 _runtime‚ 254.045601566 0loss/SRqqt2/detection_loss‚1.090827266438405 .loss/FBt2/detection_loss‚6.120491687881322 % _timestamp‚1.7843405953519766e+09 _step‚23 0loss/FRt2/assignment_loss‚5.2855938806962435 2loss/SRqqt2/assignment_loss‚1.6100295769728823 1loss/FBt1/assignment_loss‚0.18827747634847813 .loss/FRt2/detection_loss‚1.441835538436372 .loss/FBt1/detection_loss‚5.786638126119547:òâ^’‡‚(:‡ ôÀë҉ξ“ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ig!k´ˆ‚(:© ôÀëÒÂìÏ– disk.sda3.out‚5.734375 disk.sda2.in‚0# proc.memory.rssMB‚ 2632.94140625* proc.memory.percent‚16.070199012756348 disk.sda1.in‚0 disk.dm-1.out‚5.734375 disk.dm-0.in‚ 499.61328125$ memory_percent‚84.68434810638428 cpu‚26.27440013625658 disk.sda3.in‚0) proc.memory.availableMB‚ 2509.31640625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 499.61328125" disk.dm-0.out‚1360.44775390625 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚339927 network.recv‚112085 disk.rbd0.in‚0 disk.sda.in‚0* disk./.usagePercent‚40.029138111927644% disk.nvme0n1.out‚1360.44775390625 disk.dm-1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚1360.44775390625 disk.sda.out‚5.734375$ disk./.usageGB‚596.4630851745605 disk.nvme0n1.in‚ 499.61328125 disk.rbd1.out‚0.6875 disk.drbd2165.in‚0ÎA`‚“‰‚(:ˆ ƒÁëÒãÃù• gpu.0.gpu‚69.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚231.105' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü÷¨&¸Š‚(:­ ƒÁëÒ‘Ñ…˜ disk.sda.out‚ 5.98046875' disk.nvme0c0n1.out‚1362.89501953125% disk.nvme0n1.out‚1362.89501953125 disk.rbd1.in‚0 disk.dm-0.in‚ 499.61328125 disk.sda3.out‚ 5.98046875 proc.cpu.threads‚17 network.recv‚116971$ disk./.usageGB‚596.4631881713867 disk.nvme0n1.in‚ 499.61328125# proc.memory.rssMB‚ 2632.94140625* proc.memory.percent‚16.070199012756348 cpu‚26.28128029253349" disk.nvme0c0n1.in‚ 499.61328125 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚0.6875 disk.sda1.in‚0 network.sent‚387415 disk.dm-1.in‚0" disk.dm-0.out‚1362.89501953125 disk.sda1.out‚0) disk./.usagePercent‚40.02914502413106 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚84.68441963195801 disk.dm-1.out‚ 5.98046875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 2509.3046875'Q¹’‹‚(:‡ ’ÁëÒñàõ” gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&経‚(:ª ’ÁëÒöÖÖ' disk.nvme0c0n1.out‚1373.16943359375* disk./.usagePercent‚40.029142464055724$ disk./.usageGB‚596.4631500244141 disk.sda.in‚0 disk.rbd1.out‚0.75 disk.rbd0.in‚0 disk.drbd2165.out‚0$ memory_percent‚84.68444347381592 disk.sda1.out‚0 disk.dm-1.out‚ 6.25390625 disk.dm-0.in‚ 500.4453125) proc.memory.availableMB‚ 2509.30078125 disk.sda2.out‚0% disk.nvme0n1.out‚1373.16943359375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 500.4453125 disk.sda3.out‚ 6.25390625 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 500.4453125 proc.cpu.threads‚17 disk.sda.out‚ 6.25390625 disk.dm-1.in‚0 cpu‚26.290407994165616 network.recv‚120774" disk.dm-0.out‚1373.16943359375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2632.9453125* proc.memory.percent‚16.070222854614258 network.sent‚392876 disk.sda3.in‚0 disk.rbd1.in‚0Î~ÿC0+ _wandbruntime‚293  _runtime‚293èëð¢ÂŽ¼ $ _timestamp‚1.784340635172775e+09 _runtime‚ 293.86632767 _step‚24 & lr-AdamW/pg1‚0.0003918196324688942 & lr-AdamW/pg2‚0.0003918196324688942 trainer/global_step‚349XbáiÆÀ & lr-AdamW/pg1‚0.0003918196324688942 & lr-AdamW/pg2‚0.0003918196324688942 trainer/global_step‚349 $ _timestamp‚1.784340635172775e+09 _runtime‚ 293.86632767 _step‚24Á ˆ0+ _wandbruntime‚293  _runtime‚293Zv„³'‘­O UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11824044361728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3187866 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _runtime‚ 293.867551872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12184401008128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 354906050116 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51314001932288 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚616631 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51669265570126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 355800798208 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 360267889664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3178240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843406351738298e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5862928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2675062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51570184314880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1353973 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51314018971648 _step‚25“Ù3W‚’±O SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2675062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51570184314880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1353973 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51314018971648 _step‚25 UGDeviceÌœiŽ5$StatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11824044361728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3187866 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _runtime‚ 293.867551872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12184401008128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 354906050116 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51314001932288 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚616631 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51669265570126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 355800798208 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 360267889664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3178240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843406351738298e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5862928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77†£Íç0“+ _wandbruntime‚294  _runtime‚294MÓ ½&”·M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 % _timestamp‚1.7843406357891824e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 361278860800 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97138176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52074255816192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51717442383872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 356813432320 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2682694 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 trainer/global_step‚349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 361375998976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 356257652312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚97138176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52074255816192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51717442383872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 356813432320 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2682694 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 trainer/global_step‚349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 361375998976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 356257652312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 406496300032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 _step‚29 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3642318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 334.329993018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58932235796480 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99242496 % _timestamp‚1.7843406756360776e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139068800051209! >·'¤±O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 _step‚29 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3642318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 334.329993018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58932235796480 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99242496 % _timestamp‚1.7843406756360776e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 406155140608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 406496300032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360»Ä&0¥+ _wandbruntime‚334  _runtime‚3343”ž;ò%¦µM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3651384 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3064292 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406533590746 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 334.94479357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13941347505664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚705623 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843406762510424e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _step‚30 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.a‚ZÐÉllocated‚59192937376340 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088Ð[¿&§¹M F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13941347505664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚705623 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843406762510424e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _step‚30 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3651384 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3064292 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406533590746 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 334.94479357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¡³àñ0¨+ _wandbruntime‚334  _runtime‚334ò$:þÿ©ù 1loss/SRqqt1/detection_loss‚2.6021920244104626 /loss/FBt1/detection_loss‚2.4042486196017987 trainer/global_step‚399 /loss/FRt2/assignment_loss‚3.580953116352494 2loss/SRqqt2/assignment_loss‚1.4458516338581284 0loss/FBt1/assignment_loss‚0.2240857830097713 _runtime‚ 334.946595711 _step‚31 2loss/SRqqt1/assignment_loss‚1.6594467952356233 /loss/FRt1/detection_loss‚0.4400871558340584 .loss/FRt2/detection_loss‚1.324476255355802 epoch‚0 % _timestamp‚1.7843406762533107e+09 /loss/FRt1/assignment_loss‚4.076573882210121 1loss/FBt2/assignment_loss‚0.12737518091673206 1loss/SRqqt2/detection_loss‚0.8834074088920019 .loss/FBt2/detection_loss‚6.550877323174396 %loss/total_loss‚25.31957517885139àïHXªý _runtime‚ 334.946595711 _step‚31 2loss/SRqqt1/assignment_loss‚1.6594467952356233 /loss/FRt1/detection_loss‚0.4400871558340584 .loss/FRt2/detection_loss‚1.324476255355802 epoch‚0 % _timestamp‚1.7843406762533107e+09 /loss/FRt1/assignment_loss‚4.076573882210121 1loss/FBt2/assignment_loss‚0.12737518091673206 1loss/SRqqt2/detection_loss‚0.8834074088920019 .loss/FBt2/detection_loss‚6.550877323174396 %loss/total_loss‚25.31957517885139 1loss/SRqqt1/detection_loss‚2.6021920244104626 /loss/FBt1/detection_loss‚2.4042486196017987 trainer/global_step‚399 /loss/FRt2/assignment_loss‚3.580953116352494 2loss/SRqqt2/assignment_loss‚1.4458516338581284 0loss/FBt1/assignment_loss‚0.2240857830097713&0–.«‚(:’ ÎÁëÒ„à” gpu.0.gpu‚33.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ…¢¨¬‚(: ÎÁëÒ»¡– disk.rbd0.out‚0 disk.rbd1.out‚0.9375 disk.drbd2165.out‚0 disk.sda.in‚0 network.recv‚136174 disk.sda2.in‚0$ memory_percent‚84.79089736938477' disk.nvme0c0n1.out‚1403.72412109375 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚7.375" disk.nvme0c0n1.in‚ 630.89453125) proc.memory.percent‚16.17577075958252 disk.nvme0n1.in‚ 630.89453125 disk.dm-0.in‚ 630.89453125 disk.rbd0.in‚0* disk./.usagePercent‚40.029166528763945# disk./.usageGB‚596.463508605957% disk.nvme0n1.out‚1403.72412109375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2650.23828125 disk.sda3.out‚7.375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 network.sent‚456220" disk.dm-0.out‚1403.72412109375 disk.sda.out‚7.375' proc.memory.availableMB‚ 2491.859375 cpu‚26.386991809661048®¹ÊJ­bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÈXåÕJ®bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÑóvH¯bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP€u;N°bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPèþíJ±bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPO-UíN²bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPáwJ³bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¤X´H´bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPñüGü;µb6 jet/accuracy_4_of_4"trainer/global_step2:JP>Mâ J¶bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‹UgN·bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPØmb)J¸bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPðB0¦J¹bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP@Þ0‰HºbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPB'6ZJ»bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP²H J¼bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÊ™è;H½bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPç_tl;¾b6 jet/accuracy_1_of_2"trainer/global_step2:JPHTšJ¿bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPçŒþÔ@Àb; particle/accuracy_5_of_5"trainer/global_step2:JPßë4›NÁbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPðe”®JÂbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPàÛªHÃbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPù©ùNÄbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPõÙÒ‚HÅbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPR@Ïb; particle/accuracy_4_of_4"trainer/global_step2:JPI9‘@Ðb; particle/accuracy_4_of_5"trainer/global_step2:JP‚ZÝHÑbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPM–€iHÒbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPë~ 9JÓbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPqxòNÔbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP:M«JÕbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPNÖbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP÷á',JÿbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP5•SN€bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPhò;¦HbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPëì’¬N‚bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÂ>QNƒbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPE¦Z–H„bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP,8N…bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP݈-2H†bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPUÝìH‡bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP`3DNˆbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPù’^AJ‰bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP\–à~JŠbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP w©RH‹bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPñ]ÛÓJŒbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP”ý5JbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPNv×JŽbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP·ÆæßNbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÛÚ9JbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP`LïWN‘bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPIáÚ[H’bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÖgj“J“bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÈâ}EH”bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP)©1TH•bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPWÁÁ J–bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]˺©G—bB validation_average_jet_accuracy"trainer/global_step2:JPâP N˜bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPPC¯ÌJ™bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP=ë¡NšbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP-i%kJ›bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPØ.ÇNœbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPI/¢‡JbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP0xшJžbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP$Ä,ÆNŸbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPæÁ¥Ý; b6 jet/accuracy_2_of_2"trainer/global_step2:JPTØ BH¡bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPªV1YN¢bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP˜N(H£bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPœ¥‹J¤bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¢n4‰H¥bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPVe"J¦bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP(†2(H§bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP~‹ŒJ¨bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÉàéôH©bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¡úOšHªbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPŽ\ºJ«bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPRËt4@¬b; particle/accuracy_1_of_2"trainer/global_step2:JP•°EÖJ­bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP6ÃWÝJ®bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP>$‘ÑJ¯bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP&`ÅÂH°bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP…ÕH±bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPJ½6ôH²bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP4nÏ9³b4 particle/accuracy"trainer/global_step2:JP²¾eH´bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP:}THµbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP\æ@¶b; particle/accuracy_3_of_5"trainer/global_step2:JPµ!pN·bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP‚“eÒJÏbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÂbkNÐbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP„s;NÑbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP))­JÒbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP3àšöHÓbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP™µªNÔbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÝ[ûJÕbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP­ÂfHÖbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP ´,J×bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPFø¯ïHØbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPñºYþ;Ùb6 jet/accuracy_5_of_5"trainer/global_step2:JP0…´™HÚbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPîÈrƒHÛbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¶/!ØNÜbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP)Ä—æJÝbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP›b¦@Þb; particle/accuracy_3_of_3"trainer/global_step2:JP?µïz;ßb6 jet/accuracy_1_of_5"trainer/global_step2:JPœ–ïvJàbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP[JábE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPzûÇJâbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPF)³HãbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPMJÄJäbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶HHåbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPT]-QNæbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPôèËHçbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP–hú4JèbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPlõÓNébI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPŒß2×NêbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPp!ÔJëbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP^<9FJìbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP':¹JíbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP“áÙ<îb7 particle/specificity"trainer/global_step2:JP/%J 8ïb3 particle/f_score"trainer/global_step2:JPG©‹–HðbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÊ'.yJñbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPH„ÿbJòbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPúHä?NóbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP·2„OHôbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP÷%~}NõbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPSwNöbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPøÄ@:H÷bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÈ>ý7JøbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP­1C÷HùbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP4<ë‡HúbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÃô JûbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP0ÚØùHübC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP^ ðHýbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPùënHþbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP fSHÿbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP܃KH€bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPð”°ÀJbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPT‹N‚bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÃ%§¥HƒbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP"³ò™J„bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP‰’‘A@…b; particle/accuracy_1_of_1"trainer/global_step2:JP?o!H†bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP2u@&N‡bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPææ‹Þ@ˆb; particle/accuracy_2_of_2"trainer/global_step2:JPeòGoH‰bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPMÙªNŠbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPõJ‹bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPžfwHŒbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP½ äåJbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¨ÊhQ;Žb6 jet/accuracy_2_of_3"trainer/global_step2:JPdðþHbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP2à1OHbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÎ(ŠJ‘bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÈÖ¿H’bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPw~sßJ“bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP~¢Ÿ;”b6 jet/accuracy_3_of_5"trainer/global_step2:JP­tb @•b; particle/accuracy_2_of_5"trainer/global_step2:JPÑZz¹J–bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP«a¹N—bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP뜫…J˜bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP‘ý­&H™bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP^13’HšbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¹iàN›bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPë¸:HœbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPc mNbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÖGüeNžbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP³ÍHŸbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP–}J bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPœ¹J¡bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP­F þN¢bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP}zßJ£bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÏ%ÿX¤bI &Pé]§Furity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÝø¯H¥bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶n7H¦bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP]/LìJ§bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP7zv5J¨bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP=í¶µN©bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP)ä:E@ªb; particle/accuracy_1_of_5"trainer/global_step2:JPF)Â_H«bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÍöcH¬bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPý-*J­bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP!mÛ#J®bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP\UhN¯bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPµŒAõN°bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPV’ N±bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP#ª÷N²bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP$ kJ³bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPv=¢J;´b6 jet/accuracy_1_of_4"trainer/global_step2:JP&A…:HµbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP+sÔ‘J¶bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPdÞ%ñH·bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¼3!.H¸bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPcÜ>©;¹b6 jet/accuracy_3_of_3"trainer/global_step2:JPÿ„oJºbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP7L<ÂJ»bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÅIJ¼bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPå²²J½bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP5Û¶îJ¾bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPc #†H¿bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP;ï NÀbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÎɘRJÁbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPPâE+HÂbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇpJÃbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‰ oËHÄbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÑœ¹&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4027777910232544 *jet/accuracy_1_of_4‚0.5449633598327637 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06809937953948975 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.027847662568092346 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12265515327453613 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.004090060945600271 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0012998601887375116 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9656477570533752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.13262183964252472 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9941586852073669 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.22499999403953552 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99889075756073 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.14057238399982452 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09675749391317368 ;"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.018518518656492233 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08131331205368042 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.26620370149612427 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19283783435821533 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 _step‚32 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3071412444114685 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.018987970426678658 jet/accuracy_4_of_4‚0 ;"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.018518518656492233 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11146150529384613 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.13895492255687714 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07111579924821854 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06712017953395844 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16108430922031403 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.3500320315361023 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7295315265655518 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.09618066251277924 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9804435968399048 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.28125 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.02693764865398407 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.049446411430835724 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.019725501537322998 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.09959730505943298 /particle/accuracy_2_of_3‚0.9985013008117676 /particle/accuracy_4_of_4‚0.8722261786460876 particle/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936637878417969 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9958383440971375 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.025834813714027405 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.08837729692459106 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0010957265039905906 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.016779247671365738 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚ 353.207713779 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1756194829940796 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.12909743189811707 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15102584660053253 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.12232929468154907 ,jet/accuracy_3_of_4‚0.005907626356929541 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.020497674122452736 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.05435142666101456 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 *jet/accuracy_1_of_3‚0.5418818593025208 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.32087376713752747 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.19208604097366333 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.02633162960410118 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1423935741186142 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944759607315063 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.30134740471839905 ,particle/sensitivity‚0.48851776123046875 )loss/val_total_loss‚27.06657152536324 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.33004167675971985 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18450157344341278 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06326530873775482 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.10770975798368454 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.0873943343758583 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1428515911102295 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.36799025535583496 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 trainer/global_step‚399 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07045143842697144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.028255397453904152 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04862826317548752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9809228777885437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012531328247860074 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10000865906476974 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.07490552216768265 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1868125945329666 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17945054173469543 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923905730247498 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07110399007797241 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.02500000037252903 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.13214950263500214 7validation_average_jet_accuracy‚0.12878134846687317 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11031929403543472 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7618405818939209 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,jet/accuracy_2_of_2‚0.046111349016427994 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.027631709352135658 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.761813223361969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.1569223254919052 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938597679138184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9096976518630981 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1279803067445755 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9828238487243652 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.08458050340414047 (particle/accuracy‚0.7896437644958496 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 particle/accuracy_3_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9913429021835327 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.19084566831588745 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.02693764865398407 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5497220158576965 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.682762622833252 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.06944943964481354 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.1137377992272377 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09743662178516388 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.149685338139534 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.35981327295303345 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9781745672225952 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.15043795108795166 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.055160410702228546 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08119450509548187 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9975841045379639 jet/accuracy_1_of_5‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.001330891391262412 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1901329755783081 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20051391422748566 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06856891512870789 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12293273210525513 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.003163081593811512 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1901329755783081 +particle/specificity‚0.8719944953918457 'particle/f_score‚0.4993552267551422 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9819629192352295 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.24794279038906097 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.09531368315219879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1117735281586647 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.047423772513866425 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9959459900856018 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0442708358168602 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17020286619663239 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09073629230260849 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983875751495361 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18965284526348114 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9973836541175842 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_2_of_2‚0.9998962879180908 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.009259259328246117 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.20101386308670044 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.020020682364702225 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.11347812414169312 +jet/accuracy_2_of_3‚0.12626631557941437 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993584156036377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09770311415195465 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985566735267639 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.15408359467983246 jet/accuracy_3_of_5‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1423935741186142 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08268219232559204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985566735267639 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985119104385376 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.978769838809967 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005952381179668009 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4027777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.055160410702228546 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843406945135214e+09 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.003283863188698888 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.04023216664791107!áï(65Ï { & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99889075756073 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.14057238399982452 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09675749391317368 ;"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.018518518656492233 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08131331205368042 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.26620370149612427 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19283783435821533 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 _step‚32 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3071412444114685 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.018987970426678658 jet/accuracy_4_of_4‚0 ;"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.018518518656492233 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11146150529384613 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.13895492255687714 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07111579924821854 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06712017953395844 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16108430922031403 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.3500320315361023 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7295315265655518 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.09618066251277924 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9804435968399048 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.28125 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.02693764865398407 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.049446411430835724 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.019725501537322998 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.09959730505943298 /particle/accuracy_2_of_3‚0.9985013008117676 /particle/accuracy_4_of_4‚0.8722261786460876 particle/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936637878417969 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9958383440971375 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.025834813714027405 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.08837729692459106 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0010957265039905906 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.016779247671365738 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚ 353.207713779 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1756194829940796 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.12909743189811707 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15102584660053253 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.12232929468154907 ,jet/accuracy_3_of_4‚0.005907626356929541 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.020497674122452736 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.05435142666101456 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 *jet/accuracy_1_of_3‚0.5418818593025208 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.32087376713752747 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.19208604097366333 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.02633162960410118 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1423935741186142 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944759607315063 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.30134740471839905 ,particle/sensitivity‚0.48851776123046875 )loss/val_total_loss‚27.06657152536324 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.33004167675971985 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18450157344341278 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06326530873775482 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.10770975798368454 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.0873943343758583 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1428515911102295 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.36799025535583496 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 trainer/global_step‚399 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07045143842697144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.028255397453904152 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04862826317548752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9809228777885437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012531328247860074 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10000865906476974 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.07490552216768265 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1868125945329666 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17945054173469543 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923905730247498 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07110399007797241 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.02500000037252903 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.13214950263500214 7validation_average_jet_accuracy‚0.12878134846687317 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11031929403543472 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7618405818939209 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,jet/accuracy_2_of_2‚0.046111349016427994 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.027631709352135658 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.761813223361969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.1569223254919052 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938597679138184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9096976518630981 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1279803067445755 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9828238487243652 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.08458050340414047 (particle/accuracy‚0.7896437644958496 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 particle/accuracy_3_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9913429021835327 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.19084566831588745 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.02693764865398407 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5497220158576965 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.682762622833252 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.06944943964481354 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.1137377992272377 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09743662178516388 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.149685338139534 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.35981327295303345 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9781745672225952 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.15043795108795166 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.055160410702228546 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08119450509548187 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9975841045379639 jet/accuracy_1_of_5‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.001330891391262412 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1901329755783081 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20051391422748566 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06856891512870789 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12293273210525513 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.003163081593811512 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1901329755783081 +particle/specificity‚0.8719944953918457 'particle/f_score‚0.4993552267551422 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9819629192352295 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.24794279038906097 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.09531368315219879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1117735281586647 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.047423772513866425 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9959459900856018 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0442708358168602 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17020286619663239 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09073629230260849 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983875751495361 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18965284526348114 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9973836541175842 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_2_of_2‚0.9998962879180908 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.009259259328246117 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.20101386308670044 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.020020682364702225 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.11347812414169312 +jet/accuracy_2_of_3‚0.12626631557941437 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993584156036377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09770311415195465 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985566735267639 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.15408359467983246 jet/accuracy_3_of_5‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1423935741186142 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08268219232559204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985566735267639 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985119104385376 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚êª9õp0.978769838809967 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005952381179668009 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4027777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.055160410702228546 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843406945135214e+09 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.003283863188698888 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.04023216664791107 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14370334148406982 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.001815904164686799 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4027777910232544 *jet/accuracy_1_of_4‚0.5449633598327637 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06809937953948975 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.027847662568092346 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12265515327453613 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.004090060945600271 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0012998601887375116 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9656477570533752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.13262183964252472 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9941586852073669 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.22499999403953552 º[Ò*”Ђ(:‰ ÝÁëÒõûæ“ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q_±Ñ‚(:¦ ÝÁëÒ«Å• disk.sda2.out‚0 disk.dm-0.in‚ 716.50390625 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚1507.771484375# disk.nvme0n1.out‚1507.771484375$ memory_percent‚87.89036273956299 disk.nvme0n1.in‚ 716.50390625 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1984.04296875 cpu‚26.45323121531644 disk.sda3.in‚0 disk.sda3.out‚ 7.54296875) proc.memory.percent‚16.27199649810791( disk./.usagePercent‚40.0613164669289$ disk./.usageGB‚596.9425659179688 disk.rbd1.out‚0.9375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚141384% disk.nvme0c0n1.out‚1507.771484375 disk.sda.out‚ 7.54296875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 7.54296875# proc.memory.rssMB‚ 2666.00390625 proc.cpu.threads‚17 network.sent‚504123" disk.nvme0c0n1.in‚ 716.50390625 disk.sda1.in‚0š¯_’Ò‚(:‡ ìÁëÒÕµñ“ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚266.132' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´°Ó‚(:¥ ìÁëÒôÁ…– disk.sda3.out‚ 8.0078125 disk.rbd1.in‚0) proc.memory.availableMB‚ 1984.55859375 network.sent‚561554 disk.dm-1.in‚0' disk.nvme0c0n1.out‚1913.12255859375 disk.drbd2165.out‚0 disk.dm-0.in‚ 818.58203125 cpu‚26.448652195345925 proc.cpu.threads‚17 disk.rbd1.out‚1 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚1913.12255859375 disk.sda.in‚0 disk.nvme0n1.in‚ 818.58203125% disk.nvme0n1.out‚1913.12255859375" proc.memory.rssMB‚ 2666.0078125$ memory_percent‚87.88721561431885 disk.sda2.in‚0$ disk./.usageGB‚596.9424705505371 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0) disk./.usagePercent‚40.06131006674055 disk.drbd2165.in‚0) proc.memory.percent‚16.27202033996582 network.recv‚145630 disk.dm-1.out‚ 8.0078125" disk.nvme0c0n1.in‚ 818.58203125 disk.sda.out‚ 8.0078125 disk.sda2.out‚0‚F´”Ô‚(:‰ ûÁëÒ—Ò›” gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RX<›Õ‚(: ûÁëÒüÛ…– disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd1.out‚1) proc.memory.availableMB‚ 1984.30859375 disk.dm-1.out‚8.25 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚1916.8896484375! disk.dm-0.out‚1916.8896484375 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 818.58203125 disk.sda2.in‚0 disk.nvme0n1.in‚ 818.58203125 proc.cpu.threads‚17 disk.sda3.out‚8.25" proc.memory.rssMB‚ 2666.0078125) disk./.usagePercent‚40.06133694753164 disk.drbd2165.out‚0# memory_percent‚87.8887414932251 disk.rbd1.in‚0 network.recv‚149399" disk.nvme0c0n1.in‚ 818.58203125" disk./.usageGB‚596.94287109375) proc.memory.percent‚16.27202033996582 cpu‚26.445714632783208 network.sent‚566860 disk.sda.out‚8.25$ disk.nvme0n1.out‚1916.8896484375 disk.sda1.in‚0 disk.sda1.out‚0à‘k10Ö+ _wandbruntime‚394  _runtime‚394Lï$Ä×¾ & lr-AdamW/pg1‚0.0004490700173624673 & lr-AdamW/pg2‚0.0004490700173624673 trainer/global_step‚449 % _timestamp‚1.7843407362687652e+09 _runtime‚ 394.962390278 _step‚33ÍÕ ÈØ & lr-AdamW/pg1‚0.0004490700173624673 & lr-AdamW/pg2‚0.0004490700173624673 trainer/global_step‚449 % _timestamp‚1.7843407362687652e+09 _runtime‚ 394.962390278 _step‚33!0ð_ 0Ù+ _wandbruntime‚394  _runtime‚394”ó8d²'Ú¬O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843407362699466e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68223853928448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 474925038080 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92510208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 _runtime‚ 394.963679983 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4167152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558928 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558928 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843407362699466e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68223853928448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0"N`V¾0Ü+ _wandbruntime‚395  _runtime‚395'ëNíݶM B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67599360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68847049669632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚978459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4166229 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 475937672192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4176218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474869801723 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15216242460672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843407368848681e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1773855 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978219 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 475937672192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 475596326912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I@DeviceStatÏsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68020260044800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4176218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15216261072896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68495470396517 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0£fÀÀ&ÞºM PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 475937672192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 475596326912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68020260044800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4176218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15216261072896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68495470396517 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67599360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68847049669632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚978459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4166229 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 475937672192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4176218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474869801723 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15216242460672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843407368848681e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1773855 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978219 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912#á}Ý0ß+ _wandbruntime‚395  _runtime‚395¦(}Oàú 0loss/FBt1/assignment_loss‚0.4196616121375331 0loss/FBt2/assignment_loss‚0.1953688002598686 1loss/SRqqt1/detection_loss‚2.1339869298495184 _step‚36 0loss/FRt2/assignment_loss‚3.1033625761996286 /loss/FBt2/detection_loss‚3.5258959094771485 epoch‚1 trainer/global_step‚449 % _timestamp‚1.7843407368876107e+09 _runtime‚ 395.581563256 2loss/SRqqt2/assignment_loss‚1.4663449601017142 /loss/FRt1/detection_loss‚0.4861729284028705 /loss/FRt2/detection_loss‚1.3015680307627395 .loss/FBt1/detection_loss‚9.268690720804523 %loss/total_loss‚28.36033179206135 /loss/FRt1/assignment_loss‚3.995881016834362 2loss/SRqqt1/assignment_loss‚1.6676532652820169 1loss/SRqqt2/detection_loss‚0.7957450419494231<Ï17áþ 0loss/FRt2/assignment_loss‚3.1033625761996286 /loss/FBt2/detection_loss‚3.5258959094771485 epoch‚1 trainer/global_step‚449 % _timestamp‚1.7843407368876107e+09 _runtime‚ 395.581563256 2loss/SRqqt2/assignment_loss‚1.4663449601017142 /loss/FRt1/detection_loss‚0.4861729284028705 /loss/FRt2/detection_loss‚1.3015680307627395 .loss/FBt1/detection_loss‚9.268690720804523 %loss/total_loss‚28.36033179206135 /loss/FRt1/assignment_loss‚3.995881016834362 2loss/SRqqt1/assignment_loss‚1.6676532652820169 1loss/SRqqt2/detection_loss‚0.7957450419494231 0loss/FBt1/assignment_loss‚0.4196616121375331 0loss/FBt2/assignment_loss‚0.1953688002598686 1loss/SRqqt1/detection_loss‚2.1339869298495184 _step‚36$—7Ü“â‚(:ˆ ŠÂëÒòü€” gpu.0.gpu‚33.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Jô¨ã‚(: ŠÂë҉й•$ disk./.usageGB‚596.9429740905762 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 819.546875 network.recv‚153031 disk.sda.in‚0 disk.sda3.out‚8.78125 disk.rbd1.out‚1.0625 disk.dm-1.out‚8.78125 proc.cpu.threads‚17% disk.nvme0n1.out‚1929.28759765625 disk.dm-0.in‚ 819.546875( proc.memory.availableMB‚ 1983.8046875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚1929.28759765625" disk.dm-0.out‚1929.28759765625 disk.sda.out‚8.78125$ memory_percent‚87.89181709289551 cpu‚26.443056786032482) disk./.usagePercent‚40.06134385973507 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 819.546875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚572209" proc.memory.rssMB‚ 2666.0078125) proc.memory.percent‚16.27202033996582¨þ¨\’ä‚(:‡ ™ÂëÒѦɕ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öæ˜å‚(: ™ÂëÒዼ– disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 1937.19140625 proc.cpu.threads‚17$ disk./.usageGB‚596.9428787231445 disk.sda2.out‚0 disk.dm-0.in‚ 820.37890625 disk.sda.out‚9.109375 disk.nvme0n1.in‚ 820.37890625 disk.sda3.out‚9.109375 disk.dm-1.out‚9.109375$ disk.nvme0c0n1.out‚ 1937.19140625 disk.sda.in‚0 cpu‚26.440377510326552 network.recv‚157164) disk./.usagePercent‚40.06133745954671 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 820.37890625" proc.memory.rssMB‚ 2666.0078125 network.sent‚636246" disk.nvme0n1.out‚ 1937.19140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 memory_percent‚ 87.890625) proc.memory.percent‚16.27202033996582 disk.rbd1.out‚1.0625 proc.memory.availableMB‚1984 disk.rbd1.in‚0áÏ›ãžæ‚(:“ ¨ÂëÒ›Íè“ gpu.0.gpu‚42.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.867999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©áX¤ç‚(:™ ¨ÂëÒ÷눖$ disk.nvme0c0n1.out‚ 1942.30078125 network.recv‚160865$ disk./.usageGB‚596.9428901672363 disk.sda2.in‚0 disk.nvme0n1.in‚ 949.9296875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 949.9296875 disk.rbd1.out‚1.125 cpu‚26.440197421356473 proc.cpu.threads‚17 network.sent‚639606 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.27202033996582! disk.nvme0c0n1.in‚ 949.9296875 disk.dm-0.out‚ 1942.30078125 disk.sda.out‚9.484375* disk./.usagePercent‚40.061338227569316" disk.nvme0n1.out‚ 1942.30078125 disk.drbd2165.out‚0' proc.memory.availableMB‚ 1984.171875" proc.memory.rssMB‚ 2666.0078125 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚9.484375$ memory_percent‚87.88957595825195 disk.sda2.out‚0 disk.sda3.out‚9.484375 disk.sda1.out‚0 disk.dm-1.in‚0Íhÿb0è+ _wandbruntime‚435  _runtime‚435y‘­Äé¾ & lr-AdamW/pg1‚0.0004490486737098986 & lr-AdamW/pg2‚0.0004490486737098986 trainer/global_step‚499 % _timestamp‚1.7843407767527866e+09 _runtime‚ 435.446583877 _step‚37Ü7š8Èê & lr-AdamW/pg1‚0.0004490486737098986 & lr-AdamW/pg2‚0.0004490486737098986 trainer/global_step‚499 % _timestamp‚1.7843407767527866e+09 _runtime‚ 435.446583877 _step‚37%butÙ0ë+ _wandbruntime‚435  _runtime‚435H†eµ'ì¯O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75724098751319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4621604 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 525620539904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1080037 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 524815784791 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17386332474368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75199265927168 _runtime‚ 435.447788701 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚499 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843407767540574e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75585905410048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚499 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843407767540574e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75585905410048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077&Á‚Çô0î+ _wandbruntime‚436  _runtime‚436—…¾&ï¸M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚39 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3948160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚882643 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3948158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3948160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16887680387072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚499 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4630670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4620681 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76259796653056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 436.062398219 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚39 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3948160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚882643 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3948158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3948160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16887680387072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚499 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4630670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4620681 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76259796653056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 436.062398219 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 EÀ!þòø _runtime‚ 436.065345802 /loss/FRt1/assignment_loss‚3.831484522049626 0loss/SRqqt1/assignment_loss‚1.70013787005525 1loss/SRqqt2/detection_loss‚0.7476674858613492 epoch‚1 trainer/global_step‚499 0loss/FRt2/assignment_loss‚2.8484914231588907 0loss/SRqqt1/detection_loss‚1.754439079286179 %loss/total_loss‚27.51775892668452 % _timestamp‚1.7843407773720555e+09 _step‚40 /loss/FRt2/detection_loss‚1.4658097772516792 2loss/SRqqt2/assignment_loss‚1.4579347110997436 0loss/FBt1/assignment_loss‚0.6038110360255814 1loss/FBt2/assignment_loss‚0.16173484705555816 /loss/FRt1/detection_loss‚0.5926512868450876 /loss/FBt1/detection_loss‚10.706131386756898 .loss/FBt2/detection_loss‚1.647465501238676A±Í%óü _step‚40 /loss/FRt2/detection_loss‚1.4658097772516792 2loss/SRqqt2/assignment_loss‚1.4579347110997436 0loss/FBt1/assignment_loss‚0.6038110360255814 1loss/FBt2/assignment_loss‚0.16173484705555816 /loss/FRt1/detection_loss‚0.5926512868450876 /loss/FBt1/detection_loss‚10.706131386756898 .loss/FBt2/detection_loss‚1.647465501238676 _runtime‚ 436.065345802 /loss/FRt1/assignment_loss‚3.831484522049626 0loss/SRqqt1/assignment_loss‚1.70013787005525 1loss/SRqqt2/detection_loss‚0.7476674858613492 epoch‚1 trainer/global_step‚499 0loss/FRt2/assignment_loss‚2.8484914231588907 0loss/SRqqt1/detection_loss‚1.754439079286179 %loss/total_loss‚27.51775892668452 % _timestamp‚1.7843407773720555e+09(ñõ„˜”ô‚(:‰ ·ÂëÒ•­•• gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ͪ¶õ‚(:« ·ÂëҞŜ–) disk./.usagePercent‚40.06129393826589 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 950.89453125" disk.dm-0.out‚1950.46435546875 disk.sda3.in‚0 disk.sda3.out‚ 9.81640625$ memory_percent‚87.89248466491699 disk.sda.in‚0 disk.sda.out‚ 9.81640625 disk.sda2.in‚0 disk.rbd1.out‚1.125 disk.rbd0.out‚0( proc.memory.availableMB‚ 1983.6953125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2666.0078125 cpu‚26.437876299441506 disk.nvme0n1.in‚ 950.89453125 proc.cpu.threads‚17 network.sent‚644931 disk.dm-1.out‚ 9.81640625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚1950.46435546875 disk.dm-0.in‚ 950.89453125 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚164752 disk.sda2.out‚0% disk.nvme0n1.out‚1950.46435546875 disk.sda1.out‚0$ disk./.usageGB‚596.9422302246094 disk.rbd0.in‚0) proc.memory.percent‚16.27202033996582gˆû’ö‚(:‡ ÆÂëÒ‘£ã” gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÈd|·÷‚(:¬ ÆÂëÒɳז disk.nvme0n1.in‚ 951.7265625" proc.memory.rssMB‚ 2666.0078125) disk./.usagePercent‚40.06130238651453$ disk./.usageGB‚596.9423561096191% disk.nvme0n1.out‚1960.27685546875 disk.dm-1.out‚ 10.14453125 disk.sda2.out‚0 disk.sda3.out‚ 10.14453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚90.89343547821045 disk.rbd1.out‚1.1875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 951.7265625 network.sent‚708029 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1960.27685546875) proc.memory.availableMB‚ 1492.01953125) proc.memory.percent‚16.27202033996582 proc.cpu.threads‚17 network.recv‚169081 disk.drbd2165.in‚0" disk.dm-0.out‚1960.27685546875 disk.sda.in‚0 disk.sda.out‚ 10.14453125 disk.rbd1.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 951.7265625 disk.sda3.in‚0 cpu‚26.43667456274072–8S0ø+ _wandbruntime‚476  _runtime‚476ÃwÁÆùÀ _runtime‚ 476.098330393 _step‚41 ' lr-AdamW/pg1‚0.00044901291018656245 ' lr-AdamW/pg2‚0.00044901291018656245 trainer/global_step‚549 % _timestamp‚1.7843408174047601e+09®3+àÊúÄ ' lr-AdamW/pg1‚0.00044901291018656245 ' lr-AdamW/pg2‚0.00044901291018656245 trainer/global_step‚549 % _timestamp‚1.7843408174047601e+09 _runtime‚ 476.098330393 _step‚41)óÜ” )û+ _wandbruntime‚476  _runtim½ù”8e‚476@ Fµ'ü¯O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 _step‚42 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264ýì3¯¹'ý³O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 _step‚42 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9414882 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19143470651904 _runtime‚ 476.714723116 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 _step‚43 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚549 % _timestamp‚1.7843408180209725e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 576102775539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 _step‚43 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚549 % _timestamp‚1.7843408180209725e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 576102775539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9414882 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19143470651904 _runtime‚ 476.714723116 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437+‡—õ 0+ _wandbruntime‚476  _runtime‚4761@Fâ‚þ epoch‚1 0loss/FRt2/assignment_loss‚2.7717090203492285 2loss/SRqqt1/assignment_loss‚1.5302296931409651 0loss/FBt1/assignment_loss‚0.6071827413096237 /loss/FRt1/detection_loss‚0.6166524190911659 1loss/SRqqt1/detection_loss‚1.8243002449669639 1loss/SRqqt2/detection_loss‚0.7243311337844042 0loss/FRt1/assignment_loss‚3.7298206747852425 2loss/SRqqt2/assignment_loss‚1.4127610762939926 trainer/global_step‚549 _step‚44 /loss/FBt2/detection_loss‚1.9650773263819665 /loss/FRt2/detection_loss‚1.3165979951623483 /loss/FBt1/detection_loss‚1.9277421516530655 _runtime‚ 476.717540093 1loss/FBt2/assignment_loss‚0.10661703870851943 &loss/total_loss‚18.533021515627485 % _timestamp‚1.7843408180242283e+09ÿêÇŽƒ‚ /loss/FBt1/detection_loss‚1.9277421516530655 _runtime‚ 476.717540093 1loss/FBt2/assignment_loss‚0.10661703870851943 &loss/total_loss‚18.533021515627485 % _timestamp‚1.7843408180242283e+09 epoch‚1 0loss/FRt2/assignment_loss‚2.7717090203492285 2loss/SRqqt1/assignment_loss‚1.5302296931409651 0loss/FBt1/assignment_loss‚0.6071827413096237 /loss/FRt1/detection_loss‚0.6166524190911659 1loss/SRqqt1/detection_loss‚1.8243002449669639 1loss/SRqqt2/detection_loss‚0.7243311337844042 0loss/FRt1/assignment_loss‚3.7298206747852425 2loss/SRqqt2/assignment_loss‚1.4127610762939926 trainer/global_step‚549 _step‚44 /loss/FBt2/detection_loss‚1.9650773263819665 /loss/FRt2/detection_loss‚1.3165979951623483,ïÊ?ý’„‚(:‡ ÕÂëÒÔóŒ” gpu.0.gpu‚84.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˸×+±…‚(:¦ ÕÂë҆ݠ– disk.sda3.out‚ 10.38671875 disk.sda.out‚ 10.38671875$ memory_percent‚90.89696407318115) proc.memory.availableMB‚ 1491.44140625" proc.memory.rssMB‚ 2666.0078125 network.sent‚713382 disk.rbd0.in‚0 disk.dm-0.out‚1963.091796875) proc.memory.percent‚16.27202033996582) disk./.usagePercent‚40.06134565178781 disk.sda3.in‚0 disk.rbd0.out‚0 network.recv‚173041 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 951.7265625% disk.nvme0c0n1.out‚1963.091796875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚1.1875 disk.sda1.in‚0 disk.dm-1.out‚ 10.38671875 disk.dm-0.in‚ 951.7265625 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 951.7265625# disk./.usageGB‚596.943000793457 disk.sda2.out‚0# disk.nvme0n1.out‚1963.091796875 cpu‚26.433673170522194ÀU•ž†‚(:“ äÂëÒš¨€” gpu.0.gpu‚92.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›H툶‡‚(:« äÂëÒ˜ÜÕ disk.sda1.out‚0) proc.memory.availableMB‚ 1491.41015625) disk./.usagePercent‚40.06135358802137 disk.dm-1.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚1973.8330078125 disk.sda.out‚ 10.5546875 disk.sda2.in‚0! disk.nvme0n1.in‚ 1123.47265625$ disk.nvme0n1.out‚1973.8330078125 disk.sda3.out‚ 10.5546875 disk.rbd1.in‚0 disk.rbd1.out‚1.25 disk.rbd0.in‚0 disk.dm-1.out‚ 10.5546875! disk.dm-0.out‚1973.8330078125 proc.cpu.threads‚17$ memory_percent‚90.89715480804443 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1123.47265625 disk.dm-0.in‚ 1123.47265625 disk.sda.in‚0" proc.memory.rssMB‚ 2666.0078125 network.sent‚776403$ disk./.usageGB‚596.9431190490723 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚177879 disk.rbd0.out‚0) proc.memory.percent‚16.27202033996582 cpu‚26.431831121992335n× Ÿˆ‚(:” óÂëÒ÷Ôç“ gpu.0.gpu‚86.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02u-»‰‚(:° óÂëÒòóð•$ disk./.usageGB‚596.9428977966309# memory_percent‚90.8973217010498" disk.dm-0.out‚1978.38232421875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚1978.38232421875 disk.sda3.in‚0! disk.nvme0n1.in‚ 1123.47265625% disk.nvme0n1.out‚1978.38232421875 disk.sda1.in‚0 disk.dm-1.out‚ 10.97265625 disk.drbd2165.out‚0 cpu‚26.431372335350765 proc.cpu.threads‚17 disk.dm-0.in‚ 1123.47265625# disk.nvme0c0n1.in‚ 1123.47265625( proc.memory.availableMB‚ 1491.3828125 network.recv‚181938 disk.sda.out‚ 10.97265625 disk.sda2.out‚0 disk.rbd1.out‚1.25 network.sent‚781748 disk.rbd1.in‚0) proc.memory.percent‚16.27202033996582 disk.sda3.out‚ 10.97265625* disk./.usagePercent‚40.061338739584386 disk.sda2.in‚0" proc.memory.rssMB‚ 2666.0078125x5ª0Š+ _wandbruntime‚516  _runtime‚516èïv±Ä‹¾ & lr-AdamW/pg2‚0.0004489627290894832 trainer/global_step‚599 % _timestamp‚1.7843408577548127e+09 _runtime‚ 516.448537368 _step‚45 & lr-AdamW/pg1‚0.0004489627290894832€’ÂȌ trainer/global_step‚599 % _timestamp‚1.7843408577548127e+09 _runtime‚ 516.448537368 _step‚45 & lr-AdamW/pg1‚0.0004489627290894832 & lr-AdamW/pg2‚0.0004489627290894832-8+o0+ _wandbruntime‚516  _runtime‚516ÎÕî=»'ŽµO OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90937036956160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5520533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90476041323093 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90310008373248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89850350153728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1286279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚46 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5520533 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 trainer/global_step‚599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1055401 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843408577560368e+09öÔnk"¹O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89850350153728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1286279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚46 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5520533 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 trainer/global_step‚599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1055401 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843408577560368e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90937036956160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 E?DeviceStatsMonitor.on_trWż“Tain_batch_start/oversize_segments.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5520533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90476041323093 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90310008373248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335.ÃV-0+ _wandbruntime‚517  _runtime‚517Ýô„Å&‘¿M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 517.063933533 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90457249402880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1288594 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635597559808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91084932235264 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627682832384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89996843786240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89996860825600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 628024177664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1288330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5539574 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2345740 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90623563048687 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20866065228800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20230574852096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 627059812441 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843408583701885e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585ðJÿ¥É&’ÃM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91085290619904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5529585 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _step‚47 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90457266442240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90457249402880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 626719262447 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20230574852096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 627059812441 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843408583701885e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 517.063933533 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90457249402880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1288594 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635597559808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91084932235264 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627682832384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89996843786240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89996860825600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 628024177664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1288330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5539574 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2345740 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90623563048687 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20866065228800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 O𾛂(:³  ÃëÒœúØ–) proc.memory.availableMB‚ 1491.49609375 network.sent‚855548 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 11.61328125) proc.memory.percent‚16.27204418182373 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2003.51513671875 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚90.89663028717041) disk./.usagePercent‚40.06132286711726 disk.dm-1.in‚0 disk.dm-1.out‚ 11.61328125" disk.dm-0.out‚2003.51513671875 disk.sda.out‚ 11.61328125% disk.nvme0n1.out‚2003.51513671875 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1279.48828125 disk.drbd2165.out‚0 disk.dm-0.in‚ 1279.48828125 network.recv‚194026 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0$ disk./.usageGB‚596.9426612854004 disk.sda2.out‚0 disk.rbd1.out‚1.4375# proc.memory.rssMB‚ 2666.01171875 cpu‚26.42887677350595! disk.nvme0n1.in‚ 1279.48828125/´ö 0œ+ _wandbruntime‚556  _runtime‚5563%±Á» % lr-AdamW/pg1‚0.000448898133641699 % lr-AdamW/pg2‚0.000448898133641699 trainer/global_step‚649 % _timestamp‚1.7843408982577922e+09 _runtime‚ 556.95142286 _step‚49ºÏ „Åž¿ % lr-AdamW/pg1‚0.000448898133641699 % lr-AdamW/pg2‚0.000448898133641699 trainer/global_step‚649 % _timestamp‚1.7843408982577922e+09 _runtime‚ 556.95142286 _step‚491€©}¶0Ÿ+ _wandbruntime‚556  _runtime‚556¡v |f µO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 556.952714488 _step‚50 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22554317548032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 676324695633 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5974985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21868584019968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677707045376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97175866707968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.pe€ÀK1Uak‚1169 % _timestamp‚1.7843408982589753e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 trainer/global_step‚649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97672076894208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97672059854848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 677365707264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 677365707264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 556.952714488 _step‚50 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22554317548032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 676324695633 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5974985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21868584019968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677707045376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97175866707968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843408982589753e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 trainer/global_step‚649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97672076894208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97672059854848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97819300884480 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 678378334208 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.784340898872944e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 678378334208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚649 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092958 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 686674672640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5092958 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22588771827712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98497679218688 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1144393 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97999713129195 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22588668838912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 _step‚51 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677676299349 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97819317923840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97819300884480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98498037603328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1144396 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678719679488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22588771827712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98497679218688 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1144393 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97999713129195 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22588668838912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 _step‚51 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677676299349 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97819317923840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97819300884480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98498037603328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1144396 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678719679488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97819300884480 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 678378334208 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚03£.Z0¥+ _wandbruntime‚557  _runtime‚557ß’Þ¹ÿ¦ù 0loss/FBt1/assignment_loss‚0.6146082735828323 1loss/SRqqt1/detection_loss‚1.4056360848787193 %loss/total_loss‚22.30428370811658 epoch‚1 $ _timestamp‚1.784340898876129e+09 2loss/SRqqt1/assignment_loss‚1.1669048417947794 /loss/FRt1/detection_loss‚0.6533477119751826 trainer/global_step‚649 0loss/FRt2/assignment_loss‚2.1857866936496326 2loss/SRqqt2/assignment_loss‚1.1951613847009994 /loss/FRt2/detection_loss‚1.5308362788009358 _runtime‚ 557.569451706 _step‚52 0loss/FRt1/assignment_loss‚3.2958281934585614 1loss/FBt2/assignment_loss‚0.07199294076202185 0loss/SRqqt2/detection_loss‚0.846052381458816 .loss/FBt1/detection_loss‚7.583378669193813 .loss/FBt2/detection_loss‚1.754750253860286Ýž·9§ý /loss/FRt2/detection_loss‚1.5308362788009358 _runtime‚ 557.569451706 _step‚52 0loss/FRt1/assignment_loss‚3.2958281934585614 1loss/FBt2/assignment_loss‚0.07199294076202185 0loss/SRqqt2/detection_loss‚0.846052381458816 .loss/FBt1/detection_loss‚7.583378669193813 .loss/FBt2/detection_loss‚1.754750253860286 0loss/FBt1/assignment_loss‚0.6146082735828323 1loss/SRqqt1/detection_loss‚1.4056360848787193 %loss/total_loss‚22.30428370811658 epoch‚1 $ _timestamp‚1.784340898876129e+09 2loss/SRqqt1/assignment_loss‚1.1669048417947794 /loss/FRt1/detection_loss‚0.6533477119751826 trainer/global_step‚649 0loss/FRt2/assignment_loss‚2.1857866936496326 2loss/SRqqt2/assignment_loss‚1.19516138470099944X MŸ¨‚(:” ¯ÃëÒ»À„” gpu.0.gpu‚74.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¹‚É´©‚(:© ¯ÃëҥЋ– network.sent‚860982* disk./.usagePercent‚40.061337203539175$ disk./.usageGB‚596.9428749084473 disk.nvme0n1.in‚ 1280.3203125 disk.sda1.out‚0% disk.nvme0c0n1.out‚2011.130859375 disk.sda.in‚0 disk.sda.out‚ 11.99609375 disk.sda2.out‚0 disk.sda3.out‚ 11.99609375 disk.rbd0.out‚0 disk.dm-1.out‚ 11.99609375 disk.drbd2165.out‚0 proc.cpu.threads‚17& proc.memory.availableMB‚ 1491.59375 network.recv‚197800# disk.nvme0n1.out‚2011.130859375 disk.rbd1.in‚0 disk.rbd1.out‚1.4375 disk.dm-0.in‚ 1280.3203125 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 1280.3203125$ memory_percent‚90.89603424072266 disk.sda3.in‚0# proc.memory.rssMB‚ 2666.01171875) proc.memory.percent‚16.27204418182373 disk.dm-0.out‚2011.130859375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚26.429044291544933/Žíš”ª‚(:‰ ¾ÃëÒµ€é” gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gλ›»«‚(:° ¾ÃëÒÉ眖$ disk./.usageGB‚596.9429702758789 disk.rbd1.out‚1.5 network.recv‚203789 disk.rbd0.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚2020.57177734375 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚2020.57177734375) proc.memory.percent‚16.27204418182373# disk.nvme0c0n1.in‚ 1281.28515625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2020.57177734375 disk.sda2.out‚0! disk.nvme0n1.in‚ 1281.28515625 disk.rbd1.in‚0$ memory_percent‚90.89460372924805 disk.dm-0.in‚ 1281.28515625 cpu‚26.426693814273882 proc.cpu.threads‚17 network.sent‚924908* disk./.usagePercent‚40.061343603727536 disk.rbd0.in‚0 disk.dm-1.out‚ 12.23828125 disk.sda.out‚ 12.23828125 disk.sda2.in‚0# proc.memory.rssMB‚ 2666.01171875 disk.sda3.in‚0 disk.sda3.out‚ 12.23828125' proc.memory.availableMB‚ 1491.8281259þ±0¬+ _wandbruntime‚597  _runtime‚597ÔÖQ*Ä­¾ & lr-AdamW/pg1‚0.0004488191279920545 & lr-AdamW/pg2‚0.0004488191279920545 trainer/global_step‚699 % _timestamp‚1.7843409386288104e+09 _runtime‚ 597.322393594 _step‚53G½,Ȯ _runtime‚ 597.322393594 _step‚53 & lr-AdamW/pg1‚0.0004488191279920545 & lr-AdamW/pg2‚0.0004488191279920545 trainer/global_step‚699 % _timestamp‚1.7843409386288104e+095–u 0¯+ _wandbruntime‚597  _runtime‚597a¼–³Æ'°ÀO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1492915 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105762172545536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6439412 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105034111336448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105228699073351 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105228341484109 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23540021946368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117676032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105034128375808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1229901 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843409386300473e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24276801312256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466926 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728061209088 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105034111336448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105228699073351 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105228341484109 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23540021946368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117676032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105034128375808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1229901 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843409386300473e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24276801312256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466926 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728061209088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 238425210886™N–Á0²+ _wandbruntime‚597  _runtime‚597Š¶~V³ËM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23573432092672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647928012800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729415181312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23573450704896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚699 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105910784586752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1495211 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729073836032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 597.939300939 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105910426202112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105181352366080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105376220799057 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5474560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6448478 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1231643 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6448478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991†=Ó]Õ&´ÏM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105910426202112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105181352366080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105376220799057 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5474560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6448478 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1231643 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6448478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23573432092672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647928012800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729415181312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23573450704896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚699 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105910784586752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1495211 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729073836032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 597.9393009397ãxˆ0µ+ _wandbruntime‚597  _runtime‚597½kVä¶û 1loss/SRqqt2/detection_loss‚0.7991605822375639 trainer/global_step‚699 % _timestamp‚1.7843409392474537e+09 2loss/SRqqt2/assignment_loss‚1.0227573557890808 /loss/FBt1/detection_loss‚12.349987694752832 %loss/total_loss‚25.76595870813044 epoch‚1 _runtime‚ 597.941120913 _step‚56 /loss/FRt1/assignment_loss‚3.296079637812472 0loss/FRt2/assignment_loss‚2.0637185133638836 2loss/SRqqt1/assignment_loss‚1.1614930002965917 .loss/FRt1/detection_loss‚0.633178769053902 /loss/FRt2/detection_loss‚1.4223278839628966 /loss/FBt2/detection_loss‚0.8868864162884603 0loss/FBt1/assignment_loss‚0.7064373579038226 1loss/FBt2/assignment_loss‚0.08572757895999258 1loss/SRqqt1/detection_loss‚1.3382039177089424Z€·ÿ _step‚56 /loss/FRt1/assignment_loss‚3.296079637812472 0loss/FRt2/assignment_loss‚2.0637185133638836 2loss/SRqqt1/assignment_loss‚1.1614930002965917 .loss/FRt1/detection_loss‚0.633178769053902 /loss/FRt2/detection_loss‚1.4223278839628966 /loss/FBt2/detection_loss‚0.8868864162884603 0loss/FBt1/assignment_loss‚0.7064373579038226 1loss/FBt2/assignment_loss‚0.08572757895999258 1loss/SRqqt1/detection_loss‚1.3382039177089424 1loss/SRqqt2/detection_loss‚0.7991605822375639 trainer/global_step‚699 % _timestamp‚1.7843409392474537e+09 2loss/SRqqt2/assignment_loss‚1.0227573557890808 /loss/FBt1/detection_loss‚12.349987694752832 %loss/total_loss‚25.76595870813044 epoch‚1 _runtime‚ 597.9411209138v””¸‚(:‰ ÍÃëÒ¨§ª• gpu.0.gpu‚72.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚205.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}[½¨¸¹‚(:­ ÍÃëÒÉ° ˜ disk.sda2.in‚0! disk.nvme0n1.in‚ 1281.28515625 disk.sda3.out‚ 12.5703125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚2024.18017578125( proc.memory.availableMB‚ 1491.5546875 disk.dm-0.in‚ 1281.28515625 disk.sda2.out‚0% disk.nvme0n1.out‚2024.18017578125 disk.rbd1.out‚1.5 disk.rbd0.in‚0$ memory_percent‚90.89627265930176) proc.memory.percent‚16.27204418182373 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 12.5703125 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) disk./.usagePercent‚40.06133592350151 disk.sda.out‚ 12.5703125 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚207347$ disk./.usageGB‚596.9428558349609# proc.memory.rssMB‚ 2666.01171875 cpu‚26.426674347533794 network.sent‚930034" disk.dm-0.out‚2024.18017578125# disk.nvme0c0n1.in‚ 1281.28515625³Fž”º‚(:‰ ÜÃëÒ¢¸•• gpu.0.gpu‚90.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06S/ܬ»‚(:¡ ÜÃëÒºüž– disk.sda3.in‚0 disk.sda3.out‚12.75 disk.sda1.in‚0 cpu‚26.42467166719274( proc.memory.availableMB‚ 1491.5234375 network.sent‚993356 disk.drbd2165.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.27204418182373* disk./.usagePercent‚40.061334899471376$ disk./.usageGB‚596.9428405761719 disk.rbd1.out‚1.5625 disk.rbd0.out‚0 disk.dm-1.out‚12.75# disk.nvme0c0n1.in‚ 1394.13671875! disk.nvme0n1.in‚ 1394.13671875 proc.cpu.threads‚17 disk.dm-0.in‚ 1394.13671875 disk.sda1.out‚0 disk.sda.out‚12.75% disk.nvme0n1.out‚2035.57861328125$ memory_percent‚90.89646339416504 network.recv‚212939 disk.dm-1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚2035.57861328125 disk.sda.in‚0# proc.memory.rssMB‚ 2666.01171875 disk.rbd1.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚2035.57861328125 disk.sda2.out‚0w€¬Ë”¼‚(:‰ ëÃëÒžõ®“ gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚207.692' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OfÒ¹½‚(:® ëÃëÒ®”š˜ disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 13.05859375) proc.memory.availableMB‚ 1491.27734375 disk.nvme0n1.in‚ 1395.1015625 disk.sda3.in‚0 network.sent‚998385 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1395.1015625 disk.dm-0.in‚ 1395.1015625% disk.nvme0n1.out‚2044.22900390625! proc.memory.rssMB‚ 2666.015625 network.recv‚216821 disk.rbd1.out‚1.5625 disk.dm-1.out‚ 13.05859375' disk.nvme0c0n1.out‚2044.22900390625 disk.sda.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚596.9429016113281 disk.rbd0.in‚0" disk.dm-0.out‚2044.22900390625 disk.sda2.out‚0$ memory_percent‚90.89796543121338 cpu‚26.42470643049043) disk./.usagePercent‚40.06133899559192 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 13.05859375) proc.memory.percent‚16.27206802368164èF‹–0¾+ _wandbruntime‚637  _runtime‚637Q@èVÆ¿À trainer/global_step‚749 % _timestamp‚1.7843409792427528e+09 _runtime‚ 637.936361945 _step‚57 ' lr-AdamW/pg1‚0.00044872571721493525 ' lr-AdamW/pg2‚0.00044872571721493525Và‡ÊÀÄ ' lr-AdamW/pg1‚0.00044872571721493525 ' lr-AdamW/pg2‚0.00044872571721493525 trainer/global_step‚749 % _timestamp‚1.7843409792427528e+09 _runtime‚ 637.936361945 _step‚579SxGI0Á+ _wandbruntime‚637  _runtime‚637©eAàÆ'ÂÀO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779098049024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25999548119040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843409792438958e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6893864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 _step‚58 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2913479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113174919528960 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778756710912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777898219331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1596328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5848526 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 637.937616603 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 788010153984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777557669449 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408ÑÝ™*;ÃÄO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2913479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113174919528960 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778756710912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777898219331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_traiÍÃ!n_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1596328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5848526 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 637.937616603 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 788010153984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚749 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777557669449 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779098049024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25999548119040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843409792438958e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6893864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 _step‚58 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872:¼Ó‹0Ä+ _wandbruntime‚638  _runtime‚638äÒCÑ&ÅËM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6902930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 638.552911965 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113323531570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1598378 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112543403847680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 % _timestamp‚1.7843409798591993e+09 _step‚59 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 trainer/global_step‚749 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26033897808384 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5856158 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 780110683136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2917274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1598110 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112752370879565 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112543420887040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111973461606400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5856160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25244870019072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0NHÕ&ÆÏM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112543403847680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 % _timestamp‚1.7843409798591993e+09 _step‚59 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 trainer/global_step‚749 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26033897808384 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5856158 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 780110683136 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2917274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1598110 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112752370879565 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112543420887040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111973461606400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5856160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25244870019072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6902930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 638.552911965 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113323531570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1598378;Î60Ç+ _wandbruntime‚638  _runtime‚638ä”aÈý 0loss/FRt1/assignment_loss‚3.1657288798987233 2loss/SRqqt1/assignment_loss‚1.1076111534534785 1loss/FBt2/assignment_loss‚0.08847334981373178 1loss/SRqqt2/detection_loss‚0.7975124485523577 /loss/FBt1/detection_loss‚2.5023966040462255 /loss/FBt2/detection_loss‚0.9637521655361214 2loss/SRqqt2/assignment_loss‚0.9051168181947068 0loss/FBt1/assignment_loss‚0.3930006064138769 &loss/total_loss‚15.271686273510717 $ _timestamp‚1.784340979861732e+09 _step‚60 /loss/FRt2/detection_loss‚1.4407191677764057 0loss/FRt2/assignment_loss‚1.8136340725421904 /loss/FRt1/detection_loss‚0.6455345105658129 1loss/SRqqt1/detection_loss‚1.4482064967170853 epoch‚1 trainer/global_step‚749 _runtime‚ 638.555005405…örÉ /loss/FRt2/detection_loss‚1.4407191677764057 0loss/FRt2/assignment_loss‚1.8136340725421904 /loss/FRt1/detection_loss‚0.6455345105658129 1loss/SRqqt1/detection_loss‚1.4482064967170853 epoch‚1 trainer/global_step‚749 _runtime‚ 638.555005405 0loss/FRt1/assignment_loss‚3.1657288798987233 2loss/SRqqt1/assignment_loss‚1.1076111534534785 1loss/FBt2/assignment_loss‚0.08847334981373178 1loss/SRqqt2/detection_loss‚0.7975124485523577 /loss/FBt1/detection_loss‚2.5023966040462255 /loss/FBt2/detection_loss‚0.9637521655361214 2loss/SRqqt2/assignment_loss‚0.9051168181947068 0loss/FBt1/assignment_loss‚0.3930006064138769 &loss/total_loss‚15.271686273510717 $ _timestamp‚1.784340979861732e+09 _step‚60DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27722129129984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7338341 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6230126 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26882879186944 $ _timestamp‚1.784341019825816e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27722236320256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872zŠÊ'ÕÄO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3103509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6230126 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26882879186944 $ _timestamp‚1.784341019825816e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27722236320256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 829452212736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119152484528128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119152467488768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119758214299648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698836 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 _runtime‚ 678.519538834 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120588024889856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 839249943040 _step‚62 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 829793550848 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 829452212736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27722129129984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7338341 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 Iðò8ô0Ö+ _wandbruntime‚679  _runtime‚679†ÿ¨Ñ&×ËM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7347393 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27756685799424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 829185210079 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6237760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6237760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚799 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7347393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 679.134984438 _step‚63 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840359241728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27756582810624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚799 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7347393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 679.134984438 _step‚63 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840359241728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27756582810624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 829185210079 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6237760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6237760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824? “ Ì0Ù+ _wandbruntime‚679  _runtime‚679ÆsÀDÚü 1loss/SRqqt2/detection_loss‚0.8373145507303653 % _timestamp‚1.7843410204435759e+09 2loss/SRqqt1/assignment_loss‚1.1343847363459172 0loss/FBt1/assignment_loss‚0.3875545524364418 &loss/total_loss‚14.331840037453116 _step‚64 0loss/FRt2/assignment_loss‚1.6235823219573056 1loss/SRqqt1/detection_loss‚1.2462588830423278 .loss/FBt1/detection_loss‚1.767851497816003 trainer/global_step‚799 0loss/FRt1/assignment_loss‚3.0938433477197735 2loss/SRqqt2/assignment_loss‚0.8230264581026154 1loss/FBt2/assignment_loss‚0.15201157559043962 /loss/FRt1/detection_loss‚0.6420412274169293 .loss/FBt2/detection_loss‚1.149483376798453 epoch‚1 _runtime‚ 679.136910836 /loss/FRt2/detection_loss‚1.4744875094965446zĤ>Û€ trainer/global_step‚799 0loss/FRt1/assignment_loss‚3.0938433477197735 2loss/SRqqt2/assignment_loss‚0.8230264581026154 1loss/FBt2/assignment_loss‚0.15201157559043962 /loss/FRt1/detection_loss‚0.6420412274169293 .loss/FBt2/detection_loss‚1.149483376798453 epoch‚1 _runtime‚ 679.136910836 /loss/FRt2/detection_loss‚1.4744875094965446 1loss/SRqqt2/detection_loss‚0.8373145507303653 % _timestamp‚1.7843410204435759e+09 2loss/SRqqt1/assignment_loss‚1.1343847363459172 0loss/FBt1/assignment_loss‚0.3875545524364418 &loss/total_loss‚14.331840037453116 _step‚64 0loss/FRt2/assignment_loss‚1.6235823219573056 1loss/SRqqt1/detection_loss‚1.2462588830423278 .loss/FBt1/detection_loss‚1.767851497816003@¬‘÷BŸÜ‚(:” §ÄëÒãêü• gpu.0.gpu‚53.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ01wÆÝ‚(:» §ÄëÒ¶“¼— network.recv‚231826' disk.nvme0c0n1.out‚2072.66357421875 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1560.98046875# proc.memory.rssMB‚ 2712.33203125$ disk./.usageGB‚596.9431800842285 disk.dm-0.in‚ 1560.98046875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 14.26953125$ memory_percent‚91.18173122406006 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 14.26953125 disk.sda.out‚ 14.26953125 disk.rbd1.out‚1.8125) proc.memory.availableMB‚ 1444.78515625* proc.memory.percent‚16.554760932922363 disk.drbd2165.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.089466e+06) disk./.usagePercent‚40.06135768414192 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1560.98046875" disk.dm-0.out‚2072.66357421875% disk.nvme0n1.out‚2072.66357421875 cpu‚26.468959466324705 disk.drbd2165.in‚0"ø¬ý0Þ+ _wandbruntime‚697  _runtime‚697Œ¶—C>ß½| + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15158751606941223 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9831472039222717 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.386886864900589 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22568121552467346 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35816439986228943 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9951441884040833 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3008197247982025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.482276976108551 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.40483248233795166 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916879534721375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2828654646873474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.3125530779361725 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4400986135005951 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843410385781338e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9579542279243469 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07836397737264633 +particle/sensitivity‚0.5563690066337585 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2805233597755432 7validation_average_jet_accuracy‚0.21527349948883057 _runtime‚ 697.272451776 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4641469717025757 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07270172238349915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9826321601867676 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3728296756744385 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5282282829284668 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.699508786201477 particle/accuracy_2_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17293579876422882 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5020533204078674 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9928011894226074 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2921203374862671 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259104728699 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.045338839292526245 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.376485675573349 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6666666865348816 (particle/accuracy‚0.8179253339767456 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894766211509705 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4366481304168701 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2998603582382202 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3866279721260071 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2121368795633316 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025499414652585983 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 'particle/f_score‚0.5675486922264099 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10613913834095001 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9786424040794373 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9871348142623901 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.4030345380306244 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3094145357608795 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5941037535667419 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3096221685409546 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013718504458665848 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4099087119102478 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2929159700870514 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2748710513114929 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8067873120307922 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9789772033691406 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2010791003704071 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.09978270530700684 +jet/accuracy_2_of_2‚0.09132606536149979 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1657121628522873 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007178112864494324 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.8824504613876343 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9952548146247864 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16298635303974152 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902923107147217 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4099087119102478 +particle/specificity‚0.8894574046134949 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2758099138736725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943872094154358 *jet/accuracy_2_of_3‚0.3352968394756317 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4898073077201843 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3435365557670593 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.37088122963905334 *loss/val_total_loss‚13.776294075184099 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16837845742702484 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9883809685707092 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45051369071006775 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021181700751185417 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35220783948898315 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_4‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9951441884040833 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08844711631536484 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07533106207847595 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7065004110336304 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9801571369171143 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8640183806419373 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932217597961426 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839966893196106 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1026369258761406 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08622994273900986 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_3‚0.9990025758743286 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.98829585313797 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013132383115589619 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.4008904993534088 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.3379737436771393 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3720621168613434 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.3201792240142822 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06714127957820892 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 *jet/accuracy_1_of_2‚0.6144816279411316 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09980210661888123 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07533106207847595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6936885714530945 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34685006737709045 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.45665162801742554 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17875775694847107 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9022334218025208 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_3_of_4‚0.05756431445479393 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33796295523643494 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5777022242546082 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.27896884083747864 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33806103467941284 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6823821663856506 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07556576281785965 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9898639917373657 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.30840471386909485 +jet/accuracy_1_of_1‚0.15689429640769958 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15712928771972656 /particle/accuracy_4_of_4‚0.9552609324455261 _step‚65 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.044490985572338104 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012935665436089039 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5333492755889893 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9891225099563599 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.267517626285553 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9859681129455566 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3258625864982605 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.057449664920568466 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5099999904632568 *jet/accuracy_1_of_3‚0.8066332936286926 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8552808165550232 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.39865413308143616 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35076814889907837 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9885281324386597 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.3379737436771393 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15233099460601807 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.267517626285553 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.31828171014785767 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9772569537162781 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5834285020828247 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2958389222621918 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_3‚0.05142457038164139 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5552260875701904 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.705971360206604 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.35390686988830566 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.009931244887411594 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 trainer/global_step‚799 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3166576325893402 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08740724623203278 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6338246464729309 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42299124598503113 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25027838349342346 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8289983868598938 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 *jet/accuracy_2_of_4‚0.5781123638153076w‡›"àÁ| 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4400986135005951 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843410385781338e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9579542279243469 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07836397737264633 +particle/sensitivity‚0.5563690066337585 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2805233597755432 7validation_average_jet_accuracy‚0.21527349948883057 _runtime‚ 697.272451776 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4641469717025757 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07270172238349915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9826321601867676 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3728296756744385 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5282282829284668 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.699508786201477 particle/accuracy_2_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17293579876422882 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5020533204078674 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9928011894226074 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2921203374862671 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259104728699 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.045338839292526245 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.376485675573349 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6666666865348816 (particle/accuracy‚0.8179253339767456 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894766211509705 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4366481304168701 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2998603582382202 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3866279721260071 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2121368795633316 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025499414652585983 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 'particle/f_score‚0.5675486922264099 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10613913834095001 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9786424040794373 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9871348142623901 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.4030345380306244 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3094145357608795 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5941037535667419 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3096221685409546 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013718504458665848 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4099087119102478 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2929159700870514 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2748710513114929 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8067873120307922 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9789772033691406 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2010791009ƒ¸H%.3704071 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.09978270530700684 +jet/accuracy_2_of_2‚0.09132606536149979 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1657121628522873 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007178112864494324 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.8824504613876343 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9952548146247864 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16298635303974152 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902923107147217 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4099087119102478 +particle/specificity‚0.8894574046134949 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2758099138736725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9943872094154358 *jet/accuracy_2_of_3‚0.3352968394756317 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4898073077201843 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3435365557670593 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.37088122963905334 *loss/val_total_loss‚13.776294075184099 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16837845742702484 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9883809685707092 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45051369071006775 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021181700751185417 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35220783948898315 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_4‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9951441884040833 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08844711631536484 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07533106207847595 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7065004110336304 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9801571369171143 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8640183806419373 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932217597961426 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839966893196106 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1026369258761406 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08622994273900986 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_3‚0.9990025758743286 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.98829585313797 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013132383115589619 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.4008904993534088 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.3379737436771393 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3720621168613434 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.3201792240142822 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06714127957820892 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 *jet/accuracy_1_of_2‚0.6144816279411316 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09980210661888123 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07533106207847595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6936885714530945 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34685006737709045 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.45665162801742554 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17875775694847107 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9022334218025208 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_3_of_4‚0.05756431445479393 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33796295523643494 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5777022242546082 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.27896884083747864 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33806103467941284 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6823821663856506 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07556576281785965 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9898639917373657 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.30840471386909485 +jet/accuracy_1_of_1‚0.15689429640769958 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15712928771972656 /particle/accuracy_4_of_4‚0.9552609324455261 _step‚65 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.044490985572338104 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012935665436089039 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5333492755889893 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9891225099563599 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.267517626285553 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9859681129455566 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3258625864982605 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.057449664920568466 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5099999904632568 *jet/accuracy_1_of_3‚0.8066332936286926 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8552808165550232 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.39865413308143616 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35076814889907837 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9885281324386597 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.3379737436771393 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15233099460601807 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.267517626285553 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.31828171014785767 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9772569537162781 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5834285020828247 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2958389222621918 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_3‚0.05142457038164139 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5552260875701904 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.705971360206604 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.35390686988830566 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.009931244887411594 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 trainer/global_step‚799 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3166576325893402 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08740724623203278 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6338246464729309 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42299124598503113 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25027838349342346 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8289983868598938 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9923686981201172 *jet/accuracy_2_of_4‚0.5781123638153076 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15158751606941223 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9831472039222717 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.386886864900589 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22568121552467346 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35816439986228943 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9951441884040833 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3008197247982025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.482276976108551 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.40483248233795166 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916879534721375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2828654646873474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.3125530779361725A/ÌÏy”á‚(:‰ ¶ÄëÒ•ÿÉ” gpu.0.gpu‚98.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é›ð4¨â‚(: ¶ÄëÒþ¾ì– disk.dm-0.in‚ 1561.8125 disk.dm-0.out‚ 2328.171875 disk.sda.in‚0 disk.sda3.out‚ 14.50390625 disk.rbd1.in‚0$ memory_percent‚92.70284175872803 disk.sda3.in‚0) proc.memory.availableMB‚ 1195.56640625 proc.cpu.threads‚17 network.sent‚ 1.139221e+06 disk.dm-1.out‚ 14.50390625 disk.sda2.out‚0 cpu‚26.49383149015818 disk.rbd1.out‚1.875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 14.50390625 disk.nvme0n1.in‚ 1561.8125 disk.nvme0c0n1.in‚ 1561.8125" disk.nvme0c0n1.out‚ 2328.171875 disk.nvme0n1.out‚ 2328.171875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2715.03515625) proc.memory.percent‚16.57125949859619) disk./.usagePercent‚40.07748487876139$ disk./.usageGB‚597.1834869384766 disk.rbd0.in‚0 disk.dm-1.in‚0 network.recv‚235952 disk.sda1.in‚0KêÙžã‚(:“ ÅÄëҸѱ“ gpu.0.gpu‚64.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚203.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚40.67679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á3Ž°ä‚(:¥ ÅÄëÒ¢ˆ¨–$ disk.nvme0c0n1.out‚ 2576.73828125 cpu‚26.491994573411862 network.sent‚ 1.196656e+06 disk.sda2.out‚0$ memory_percent‚92.69740581512451* proc.memory.percent‚16.567492485046387* disk./.usagePercent‚40.077422668930566 disk.sda2.in‚0" disk.nvme0n1.out‚ 2576.73828125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1196.45703125 network.recv‚240310# disk./.usageGB‚597.182559967041 disk.dm-1.out‚ 14.69921875 disk.nvme0c0n1.in‚ 1561.8125 disk.dm-0.out‚ 2576.73828125 disk.rbd1.out‚1.875 disk.sda3.out‚ 14.69921875 disk.rbd1.in‚0 disk.dm-0.in‚ 1561.8125 disk.sda.out‚ 14.69921875# proc.memory.rssMB‚ 2714.41796875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1561.8125 disk.sda1.in‚0 disk.sda1.out‚08HÖÔŸå‚(:” ÔÄëÒöÃà” gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»~ûµæ‚(:ª ÔÄëÒì‰ï•$ memory_percent‚92.69726276397705) disk./.usagePercent‚40.07743137318673! disk.nvme0n1.in‚ 1697.75390625 disk.rbd0.in‚0 disk.dm-1.out‚ 14.95703125# disk.nvme0c0n1.out‚ 2587.1796875) proc.memory.availableMB‚ 1196.48046875 proc.memory.rssMB‚ 2714.4375 network.sent‚ 1.201666e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1697.75390625* proc.memory.percent‚16.567611694335938 cpu‚26.487925339989506 disk.sda3.out‚ 14.95703125 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚243659! disk.nvme0n1.out‚ 2587.1796875 disk.sda3.in‚0 disk.dm-0.in‚ 1697.75390625 proc.cpu.threads‚17 disk.rbd1.out‚1.9375 disk.rbd0.out‚0 disk.dm-0.out‚ 2587.1796875 disk.sda.in‚0# disk./.usageGB‚597.182689666748 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 14.95703125^ŶJ0ç+ _wandbruntime‚739  _runtime‚739Ó$>yÆèÀ _runtime‚ 739.163357512 _step‚66 ' lr-AdamW/pg1‚0.00044849570520149966 ' lr-AdamW/pg2‚0.00044849570520149966 trainer/global_step‚849 % _timestamp‚1.7843410804697857e+09ú¤tVÊéÄ trainer/global_step‚849 % _timestamp‚1.7843410804697857e+09 _runtime‚ 739.163357512 _step‚66 ' lr-AdamW/pg1‚0.00044849570520149966 ' lr-AdamW/pg2‚0.00044849570520149966BD²p 0ê+ _wandbruntime‚739  _runtime‚739 D(úÆ'ëÀO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 896510368491 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 898222258688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129283532254185 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1494171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28570229213184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7873091 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 896850918377 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7863115 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1835250 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897880920064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚849 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3329418 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115ïììÄO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129283532254185 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1494171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28570229213184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7873091 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 896850918377 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7863115 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1835250 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897880920064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚849 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3329418 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 896510368491 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 898222258688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624C>„ne0í+ _wandbruntime‚739  _runtime‚739‘rÁYÑ&îËM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1495916 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129197073461248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚68 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 trainer/global_step‚849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 909692973568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129430696390533 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 897521422213 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7882157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 trainer/global_step‚849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 909692973568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129430696390533 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 897521422213 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7882157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1495916 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129197073461248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚68D1Z®0ð+ _wandbruntime‚739  _runtime‚739ZüIýñ÷ 0loss/FRt2/assignment_loss‚1.6483770432947307 1loss/FBt2/assignment_loss‚0.10980169989295618 /loss/FRt1/detection_loss‚0.6602304705557136 % _timestamp‚1.7843410810884237e+09 1loss/SRqqt2/detection_loss‚0.7317618234228955 _step‚69 1loss/SRqqt1/assignment_loss‚1.232471778064625 /loss/FBt1/assignment_loss‚0.488210814188538 1loss/SRqqt1/detection_loss‚1.3984070462068074 &loss/total_loss‚21.929220951480318 epoch‚2 trainer/global_step‚849 _runtime‚ 739.781699284 /loss/FRt1/assignment_loss‚3.138854299589958 2loss/SRqqt2/assignment_loss‚0.8436491068269615 /loss/FRt2/detection_loss‚1.4983971456995377 +loss/FBt1/detection_loss‚9.091806145651 /loss/FBt2/detection_loss‚1.0872535780865953Ž§f>òû &loss/total_loss‚21.929220951480318 epoch‚2 trainer/global_step‚849 _runtime‚ 739.781699284 /loss/FRt1/assignment_loss‚3.138854299589958 2loss/SRqqt2/assignment_loss‚0.8436491068269615 /loss/FRt2/detection_loss‚1.4983971456995377 +loss/FBt1/detection_loss‚9.091806145651 /loss/FBt2/detection_loss‚1.0872535780865953 0loss/FRt2/assignment_loss‚1.6483770432947307 1loss/FBt2/assignment_loss‚0.10980169989295618 /loss/FRt1/detection_loss‚0.6602304705557136 % _timestamp‚1.7843410810884237e+09 1loss/SRqqt2/detection_loss‚0.7317618234228955 _step‚69 1loss/SRqqt1/assignment_loss‚1.232471778064625 /loss/FBt1/assignment_loss‚0.488210814188538 1loss/SRqqt1/detection_loss‚1.3984070462068074E+–£žó‚(:“ ãÄëÒÃÂΓ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.359' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.47179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,…†Åô‚(:º ãÄëÒ’ê— network.sent‚ 1.205604e+06% disk.nvme0n1.out‚2595.61279296875 disk.sda3.out‚ 15.30078125 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 15.30078125 disk.sda1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 1700.37890625) proc.memory.availableMB‚ 1196.28515625' disk.nvme0c0n1.out‚2595.61279296875 cpu‚26.485450837644045 disk.rbd0.out‚0 disk.dm-0.in‚ 1700.37890625! proc.memory.rssMB‚ 2714.421875 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚1.9375 network.recv‚246463* disk./.usagePercent‚40.077440589457964 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚597.1828269958496 disk.dm-1.out‚ 15.30078125" disk.dm-0.out‚2595.61279296875 disk.sda2.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.567516326904297 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1700.37890625$ memory_percent‚92.69845485687256ÅJ}h”õ‚(:‰ òÄëÒêŽÛ“ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l$¿È¸ö‚(:­ òÄëÒÄë½– disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚2602.34423828125 network.recv‚250783 disk.sda1.out‚0 disk.dm-1.out‚15.59375! disk.nvme0n1.in‚ 1700.37890625 cpu‚26.481724999753638 network.sent‚ 1.268456e+06 disk.drbd2165.in‚0 disk.rbd1.out‚2( proc.memory.availableMB‚ 950.12109375* disk./.usagePercent‚40.077417548779884 disk.sda.out‚15.59375$ memory_percent‚94.20092105865479" disk.dm-0.out‚2602.34423828125 disk.sda3.in‚0# proc.memory.rssMB‚ 2714.44140625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚597.1824836730957 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1700.37890625' disk.nvme0c0n1.out‚2602.34423828125 disk.sda3.out‚15.59375 disk.dm-0.in‚ 1700.37890625 disk.sda.in‚0* proc.memory.percent‚16.567635536193848BA0÷+ _wandbruntime‚779  _runtime‚779[ ¼Åø¿ trainer/global_step‚899 $ _timestamp‚1.784341120900767e+09 _runtime‚ 779.594315004 _step‚70 ' lr-AdamW/pg1‚0.00044835911873842524 ' lr-AdamW/pg2‚0.00044835911873842524Øÿç÷Éùà _step‚70 ' lr-AdamW/pg1‚0.00044835911873842524 ' lr-AdamW/pg2‚0.00044835911873842524 trainer/global_step‚899 $ _timestamp‚1.784341120900767e+09 _runtime‚ 779.594315004F 5Ô«0ú+ _wandbruntime‚779  _runtime‚7798c€ûÀO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960026756608 _runtime‚ 779.595527303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7113994 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137360460335104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8327543 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135712197890048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 YEDeviceStatsMonitor.on_train_batch_start/alloÓ0…µcated_bytes.all.allocated‚137360818713088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3520984 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1581421 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136659324745447 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 trainer/global_step‚899 _step‚71 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584÷óîoÊ'üÄO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137360460335104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8327543 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135712197890048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137360818713088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3520984 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1581421 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136659324745447 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 trainer/global_step‚899 _step‚71 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960026756608 _runtime‚ 779.595527303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7113994 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872G5}oM”ý‚(:‰ ÅëÒͶ£” gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AîÀ¡ªþ‚(:Ÿ ÅëÒ¨ÝÛ– network.sent‚ 1.273542e+06 network.recv‚254485 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 1701.34375 disk.sda.out‚ 15.9921875 disk.sda3.in‚0& proc.memory.availableMB‚ 950.171875* proc.memory.percent‚16.567516326904297 disk.sda2.out‚0# disk.nvme0n1.out‚2610.013671875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 1701.34375$ memory_percent‚94.20061111450195 disk.dm-0.out‚2610.013671875 proc.cpu.threads‚17$ disk./.usageGB‚597.1835479736328 disk.sda2.in‚0 disk.rbd1.out‚2 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚2610.013671875) disk./.usagePercent‚40.07748897488193 disk.sda.in‚0 disk.sda3.out‚ 15.9921875! proc.memory.rssMB‚ 2714.421875 disk.nvme0n1.in‚ 1701.34375 disk.dm-1.out‚ 15.9921875 cpu‚26.480501830326375+\ à0ÿ+ _wandbruntime‚780  _runtime‚780lyØÐ&€ÊM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 949589048832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.784341121520907e+09 _runtime‚ 780.214705123 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31236070361600 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136559124942848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚72 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8326619 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 949930394624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1583163 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8336609 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121626 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 948478459119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31236070361600 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136559124942848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚72 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8326619 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 949930394624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1583163 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8336609 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121626 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 948478459119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 P0‰+ _wandbruntime‚820  _runtime‚820ÊOèÅŠ¿ trainer/global_step‚949 $ _timestamp‚1.784341161507745e+09 _runtime‚ 820.201377598 _step‚74 ' lr-AdamW/pg1‚0.00044820815669341065 ' lr-AdamW/pg2‚0.00044820815669341065çì&ÙÉ‹Ã _step‚74 ' lr-AdamW/pg1‚0.00044820815669341065 ' lr-AdamW/pg2‚0.00044820815669341065 trainer/global_step‚949 $ _timestamp‚1.784341161507745e+09 _runtime‚ 820.201377598JÛ©)0Œ+ _wandbruntime‚820  _runtime‚820a©Ç'ÁO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32924632249344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784341161508896e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1668671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495594 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 999613262336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144035832415201 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1668668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 _runtime‚ 820.202624432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32924541836800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71800320 _step‚75 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100úYVË'ŽÅO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32924632249344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784341161508896e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1668671 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495594 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 999613262336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144035832415201 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1668668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 _runtime‚ 820.202624432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32924541836800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71800320 _step‚75 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Kt"’0+ _wandbruntime‚820  _runtime‚820à’D ÐM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚76 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143921193463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7503224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚XSG’2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144921819360256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3715853 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143921176424448 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143184259194880 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843411621249764e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8781071 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143184242155520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8781071 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448ªáÛÚ&‘ÔM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1000625896448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86473216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843411621249764e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚949 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8781071 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143184242155520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8781071 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚76 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143921193463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7503224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144921819360256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3715853 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143921176424448 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143184259194880 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 _runtime‚ 860.846582168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151411982495709 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 _step‚79 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ­fóÑ' ËO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152186312679936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚150363265077248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784341202152865e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9226471 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 _step‚79 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152185954301952 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 _runtime‚ 860.846582168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151411982495709 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33617971750912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1051321398272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150509792788480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1050980052480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 _runtime‚ 861.461654435 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 trainer/global_step‚999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824Ø5•uÜ&£ÖM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 _runtime‚ 861.461654435 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 trainer/global_step‚999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151283227906048 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152334566343680 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33617971750912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1051321398272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150509792788480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1050980052480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528PÈž¬0¤+ _wandbruntime‚861  _runtime‚861ô;ak¥ú epoch‚2 /loss/FRt2/assignment_loss‚1.522038285948628 2loss/SRqqt1/assignment_loss‚0.8926108508691436 1loss/FBt2/assignment_loss‚0.11229201872099516 1loss/SRqqt1/detection_loss‚1.2090722600944708 /loss/FBt1/detection_loss‚15.146138254938455 /loss/FRt1/detection_loss‚0.6886687317694342 _runtime‚ 861.463364146 /loss/FBt2/detection_loss‚0.8987527236857591 $loss/total_loss‚26.7798172939197 /loss/FRt1/assignment_loss‚2.864545749328887 0loss/FBt1/assignment_loss‚0.5564699992376926 /loss/FRt2/detection_loss‚1.3652185343508463 trainer/global_step‚999 _step‚81 2loss/SRqqt2/assignment_loss‚0.6993124710433455 1loss/SRqqt2/detection_loss‚0.8246974139320394 % _timestamp‚1.7843412027700794e+095EÍÓ'¦þ /loss/FBt2/detection_loss‚0.8987527236857591 $loss/total_loss‚26.7798172939197 /loss/FRt1/assignment_loss‚2.864545749328887 0loss/FBt1/assignment_loss‚0.5564699992376926 /loss/FRt2/detection_loss‚1.3652185343508463 trainer/global_step‚999 _step‚81 2loss/SRqqt2/assignment_loss‚0.6993124710433455 1loss/SRqqt2/detection_loss‚0.8246974139320394 % _timestamp‚1.7843412027700794e+09 epoch‚2 /loss/FRt2/assignment_loss‚1.522038285948628 2loss/SRqqt1/assignment_loss‚0.8926108508691436 1loss/FB¹Ôò:Ýt2/assignment_loss‚0.11229201872099516 1loss/SRqqt1/detection_loss‚1.2090722600944708 /loss/FBt1/detection_loss‚15.146138254938455 /loss/FRt1/detection_loss‚0.6886687317694342 _runtime‚ 861.463364146Q9ómBŸ§‚(:” ÛÅëÒùš´“ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l·¿¨‚(:´ ÛÅëÒ’±õ•$ memory_percent‚94.20175552368164* proc.memory.percent‚16.567516326904297 cpu‚26.469986226992276) disk./.usagePercent‚40.07747719853536 disk.sda3.out‚ 17.47265625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚2657.99951171875 disk.sda.in‚0$ disk./.usageGB‚597.1833724975586 disk.rbd1.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 949.984375 proc.cpu.threads‚17 network.sent‚ 1.456072e+06 network.recv‚277954 disk.sda3.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2714.421875% disk.nvme0n1.out‚2657.99951171875 disk.rbd1.out‚2.25 disk.sda1.out‚0 disk.sda.out‚ 17.47265625 disk.sda2.in‚0! disk.nvme0n1.in‚ 2037.05078125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2037.05078125" disk.dm-0.out‚2657.99951171875 disk.sda2.out‚0 disk.dm-1.out‚ 17.47265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 2037.05078125#pcÜ”©‚(:‰ êÅëÒ†…Þ“ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ ×˜µª‚(:ª êÅëÒ—Äû•# disk./.usageGB‚597.183406829834 disk.sda.out‚17.78125 disk.sda3.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚40.07747950260316 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚26.467466444513892 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 2037.05078125% disk.nvme0c0n1.out‚2663.037109375$ memory_percent‚94.20225620269775* proc.memory.percent‚16.567635536193848 network.recv‚282352 disk.sda2.out‚0# disk.nvme0n1.out‚2663.037109375 disk.dm-1.in‚0( proc.memory.availableMB‚ 949.90234375! disk.nvme0n1.in‚ 2037.05078125 disk.dm-1.out‚17.78125# proc.memory.rssMB‚ 2714.44140625 disk.sda1.in‚0 disk.dm-0.out‚2663.037109375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚2.3125 disk.drbd2165.in‚0 network.sent‚ 1.519043e+06 disk.dm-0.in‚ 2037.05078125 disk.sda3.out‚17.78125 disk.rbd0.in‚0 disk.sda1.out‚0Š³þ䟫‚(:” ùÅëÒ„׿“ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Wúû·¬‚(:¬ ùÅëÒКڕ disk.dm-1.out‚ 17.953125" disk.nvme0c0n1.in‚ 2037.8828125 disk.dm-0.in‚ 2037.8828125 disk.sda2.in‚0 network.recv‚285687 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 17.953125$ memory_percent‚94.20244693756104 disk.sda3.in‚0 disk.rbd1.out‚2.3125( proc.memory.availableMB‚ 949.87109375 disk.drbd2165.in‚0! disk.dm-0.out‚2671.5771484375$ disk.nvme0n1.out‚2671.5771484375! proc.memory.rssMB‚ 2714.421875* disk./.usagePercent‚40.077486926821656 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚26.46639238461503 disk.sda1.out‚0 disk.sda.out‚ 17.953125* proc.memory.percent‚16.567516326904297 proc.cpu.threads‚17 network.sent‚ 1.524025e+06$ disk./.usageGB‚597.1835174560547 disk.sda1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚2671.5771484375 disk.nvme0n1.in‚ 2037.8828125 disk.rbd0.in‚0 disk.rbd1.in‚0ýžÄ0­+ _wandbruntime‚901  _runtime‚901oÙi„Ä®¾ & lr-AdamW/pg1‚0.0004478631455642953 & lr-AdamW/pg2‚0.0004478631455642953 trainer/global_step‚1049 $ _timestamp‚1.784341242489803e+09 _runtime‚ 901.185206733 _step‚82(†H›È¯Â _runtime‚ 901.185206733 _step‚82 & lr-AdamW/pg1‚0.0004478631455642953 & lr-AdamW/pg2‚0.0004478631455642953 trainer/global_step‚1049 $ _timestamp‚1.784341242489803e+09RòÏ}0°+ _wandbruntime‚901  _runtime‚901,[ê¼Ï'±ÉO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 _step‚83 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1099316866009 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9680923 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158788132576217 % _timestamp‚1.7843412424935646e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36370247422976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35255999531008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36370350418432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84383232†7s{Ó'²ÍO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 _runtime‚ 901.18724419 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9680923 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158788132576217 % _timestamp‚1.7843412424935646e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36370247422976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35255999531008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36370350418432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84383232 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 _step‚83 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1099316866009 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9699965 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158645296427008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train-©ÞÚ _batch_end/segment.large_pool.peak‚1169 _step‚84 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102016900096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 _runtime‚ 901.801340381 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158645296427008 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8266424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056ÔJB®Ý&µ×M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7843412431075573e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1099987369845 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158645279387648 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9699965 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158645296427008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚84 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102016900096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 _runtime‚ 901.801340381 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158645296427008 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8266424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2253420 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696832Tíò£0¶+ _wandbruntime‚901  _runtime‚901‘v4©·ý _step‚85 0loss/FRt1/assignment_loss‚2.6630320923985744 0loss/FRt2/assignment_loss‚1.5882131736830802 2loss/SRqqt2/assignment_loss‚0.7607012920000876 /loss/FRt1/detection_loss‚0.6662948771639848 /loss/FRt2/detection_loss‚1.3245600427129514 .loss/FBt1/detection_loss‚1.643043287027935 trainer/global_step‚1049 $ _timestamp‚1.784341243109975e+09 2loss/SRqqt1/assignment_loss‚1.0483010662327636 1loss/FBt1/assignment_loss‚0.25939552454593856 &loss/total_loss‚13.217394711139073 _runtime‚ 901.803273972 1loss/SRqqt2/detection_loss‚0.7611053367320608 epoch‚2 1loss/FBt2/assignment_loss‚0.12345972480575185 0loss/SRqqt1/detection_loss‚1.390421996815036 /loss/FBt2/detection_loss‚0.9888662970209096<šõ¸ 0loss/FRt1/assignment_loss‚2.6630320923985744 0loss/FRt2/assignment_loss‚1.5882131736830802 2loss/SRqqt2/assignment_loss‚0.7607012920000876 /loss/FRt1/detection_loss‚0.6662948771639848 /loss/FRt2/detection_loss‚1.3245600427129514 .loss/FBt1/detection_loss‚1.643043287027935 trainer/global_step‚1049 $ _timestamp‚1.784341243109975e+09 2loss/SRqqt1/assignment_loss‚1.0483010662327636 1loss/FBt1/assignment_loss‚0.25939552454593856 &loss/total_loss‚13.217394711139073 _runtime‚ 901.803273972 1loss/SRqqt2/detection_loss‚0.7611053367320608 epoch‚2 1loss/FBt2/assignment_loss‚0.12345972480575185 0loss/SRqqt1/detection_loss‚1.390421996815036 /loss/FBt2/detection_loss‚0.9888662970209096 _step‚85UÖ¿Ã‚(:” ˆÆëÒ㡬” gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Ÿ®à·º‚(:¬ ˆÆëÒÜÔ«–( proc.memory.availableMB‚ 950.12109375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.out‚2.375* disk./.usagePercent‚40.130797167729256# disk.nvme0c0n1.in‚ 2038.84765625# disk.nvme0c0n1.out‚ 3496.1015625 disk.dm-0.in‚ 2038.84765625 disk.sda.out‚ 18.10546875 disk.sda3.out‚ 18.10546875 network.recv‚288869 disk.sda2.out‚0$ memory_percent‚94.20092105865479 cpu‚26.46450511685484 disk.rbd0.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2038.84765625* proc.memory.percent‚16.567659378051758 network.sent‚ 1.528939e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 18.10546875! disk.nvme0n1.out‚ 3496.1015625" proc.memory.rssMB‚ 2714.4453125 disk.sda1.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚597.9778785705566 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-0.out‚ 3496.1015625 disk.sda3.in‚0ƒ=ž»‚(:“ —ÆëÒ¨Ìß“ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡8+jÀ¼‚(:µ —ÆëÒÙòý•$ disk./.usageGB‚597.9510688781738# disk.nvme0c0n1.in‚ 2214.27734375' disk.nvme0c0n1.out‚3500.48193359375" disk.dm-0.out‚3500.48193359375 disk.rbd1.out‚2.375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 18.2578125 disk.rbd0.out‚0# memory_percent‚94.2002534866333# proc.memory.rssMB‚ 2714.42578125 disk.sda.out‚ 18.2578125 disk.drbd2165.out‚0 cpu‚26.463516508461733 network.sent‚ 1.591987e+06* proc.memory.percent‚16.567540168762207 disk.dm-0.in‚ 2214.27734375 disk.sda.in‚0! disk.nvme0n1.in‚ 2214.27734375 disk.sda3.out‚ 18.2578125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 950.23046875 network.recv‚293284% disk.nvme0n1.out‚3500.48193359375 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚40.12899794677862µ–:Ï0½+ _wandbruntime‚941  _runtime‚941ÐdæÓž¿ _step‚86 & lr-AdamW/pg1‚0.0004476691186396147 & lr-AdamW/pg2‚0.0004476691186396147 trainer/global_step‚1099 % _timestamp‚1.7843412830987504e+09 _runtime‚ 941.792320307ñ-ÒÉ¿Ã _step‚86 & lr-AdamW/pg1‚0.0004476691186396147 & lr-AdamW/pg2‚0.0004476691186396147 trainer/global_step‚1099 % _timestamp‚1.7843412830987504e+09 _runtime‚ 941.792320307VľÂ0À+ _wandbruntime‚941  _runtime‚941…ÉŒóÓ'ÁÍO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165860089839616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1165660142080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 941.793537936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165860106878976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38093194565632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36927437457408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1151699767808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 _step‚87 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36927418845184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 $ _timestamp‚1.784341283099806e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10135375 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167011448268800 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 $ _timestamp‚1.784341283099806e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10135375 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167011448268800 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.cu^ö¦rrent‚9978 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 941.793537936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165860106878976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38093194565632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36927437457408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1151699767808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421WxtÐC0Ã+ _wandbruntime‚942  _runtime‚942‰èJÝ&Ä×M trainer/global_step‚1099 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166007347908608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36960866215936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166311804382431 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 942.408919393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166311446793073 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864àlá&ÅÛM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36960866215936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166311804382431 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 942.408919393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166311446793073 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 trainer/global_step‚1099 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166007347908608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088XÈTDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚91 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.smallÅkÁŸÜ!_pool.freed‚2459152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38598875383808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2017668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 982.392711085 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 % _timestamp‚1.7843413236989856e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10589827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4476820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1202053931008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2017671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173222141321216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9021992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173222158360576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173222158360576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10599803 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2459407 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173222158360576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173222158360576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10599803 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2459407 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚91 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38598875383808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2017668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 982.392711085 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 % _timestamp‚1.7843413236989856e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10589827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4476820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1202053931008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2017671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173222141321216 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9021992[["ô0Õ+ _wandbruntime‚983  _runtime‚983òc¡Þ&ÖØM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 trainer/global_step‚1149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2461481 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1203407903744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1218114211328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚92 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269ƒMM×ÜM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 trainer/global_step‚1149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2461481 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1203407903744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1218114211328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 QCDeviceStatsMoniJ™øzÇtor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚92 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1022.701578311 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 $ _timestamp‚1.784341364007862e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚1199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11054255 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1251166326733 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2562533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253090771456 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268258984960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1022.701578311 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 $ _timestamp‚1.784341364007862e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚1199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11054255 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1251166326733 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2562533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253090771456 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268258984960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 I,Úö95_ÉaÎ2ç- _wandbruntime‚1023  _runtime‚1023‹¸ñØÞ&èØM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 % _timestamp‚1.7843413646218104e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9411224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1023.315545015 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254103405568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181985554277376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180731450871808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 _step‚96 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180731433832448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089%§'Ôâ&éÜM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180731450871808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 _step‚96 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180731433832448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 % _timestamp‚1.7843413646218104e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9411224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1023.315545015 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254103405568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181985554277376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888`Kç-°2ê- _wandbruntime‚1023  _runtime‚1023Ä0Šëû /loss/FRt2/detection_loss‚1.3714116856439775 1loss/SRqqt2/detection_loss‚0.6653796337073048 /loss/FBt1/detection_loss‚0.8531309542330828 epoch‚2 /loss/FBt1/assignment_loss‚0.363021679769383 1loss/FBt2/assignment_loss‚0.12182852456365678 1loss/SRqqt1/detection_loss‚1.2364722027446762 /loss/FBt2/detection_loss‚0.7324067035810677 $loss/total_loss‚11.2665967797008 $ _timestamp‚1.784341364624816e+09 _step‚97 0loss/FRt2/assignment_loss‚1.1873099015274193 2loss/SRqqt1/assignment_loss‚0.9499132134678736 2loss/SRqqt2/assignment_loss‚0.7282067937590958 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚2.4176105630612703 .loss/FRt1/detection_loss‚0.639904923641991 _runtime‚1023.318125586z7'Ãìÿ 0loss/FRt1/assignment_loss‚2.4176105630612703 .loss/FRt1/detection_loss‚0.639904923641991 _runtime‚1023.318125586 /loss/FRt2/detection_loss‚1.3714116856439775 1loss/SRqqt2/detection_loss‚0.6653796337073048 /loss/FBt1/detection_loss‚0.8531309542330828 epoch‚2 /loss/FBt1/assignment_loss‚0.363021679769383 1loss/FBt2/assignment_loss‚0.12182852456365678 1loss/SRqqt1/detection_loss‚1.2364722027446762 /loss/FBt2/detection_loss‚0.7324067035810677 $loss/total_loss‚11.2665967797008 $ _timestamp‚1.784341364624816e+09 _step‚97 0loss/FRt2/assignment_loss‚1.1873099015274193 2loss/SRqqt1/assignment_loss‚0.9499132134678736 2loss/SRqqt2/assignment_loss‚0.7282067937590958 trainer/global_step‚1199aGÒ°Xí‚(:’ €ÇëÒд©“ gpu.0.gpu‚94.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  c»î‚(:° €Çëҵ©– disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 20.36328125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2715.828125! disk.nvme0n1.in‚ 2360.05859375 disk.dm-1.out‚ 20.36328125$ disk./.usageGB‚597.2141571044922 disk.dm-0.in‚ 2360.05859375! disk.dm-0.out‚3559.3466796875 disk.sda3.in‚0 disk.rbd1.out‚2.75* proc.memory.percent‚16.576099395751953 network.recv‚318344$ disk.nvme0n1.out‚3559.3466796875) disk./.usagePercent‚40.07954317933668 disk.rbd1.in‚0' proc.memory.availableMB‚ 948.4609375 network.sent‚ 1.755905e+06& disk.nvme0c0n1.out‚3559.3466796875 disk.sda.out‚ 20.36328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# memory_percent‚94.2110538482666# disk.nvme0c0n1.in‚ 2360.05859375 disk.sda.in‚0 disk.sda1.in‚0 cpu‚26.49154200959771 proc.cpu.threads‚177Ë!ë2ï- _wandbruntime‚1041  _runtime‚1041þYø%ðÂ{ 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8830059766769409 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6943564414978027 *jet/accuracy_1_of_3‚0.8944398760795593 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5832611322402954 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9021395444869995 + Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5458648800849915 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6326325535774231 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7499622106552124 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5683817863464355 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038793593645095825 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20365417003631592 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21937336027622223 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5902802348136902 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_3_of_4‚0.07392362505197525 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28774183988571167 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5603033900260925 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5651887655258179 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15438701212406158 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9856351017951965 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021938273683190346 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16992919147014618 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9940263628959656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23074877262115479 _runtime‚1041.467437962 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7426303625106812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9854729175567627 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929095506668091 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20398221909999847 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518519401550293 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929095506668091 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3710421919822693 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843413827729933e+09 *jet/accuracy_1_of_4‚0.9613498449325562 trainer/global_step‚1199 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9960276484489441 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04107537865638733 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7185032963752747 'particle/f_score‚0.6059921383857727 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.471800297498703 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9753967523574829 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5910869240760803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5808545351028442 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9537187218666077 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5290012359619141 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45841166377067566 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9827951788902283 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6780819892883301 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666567325592 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5836498737335205 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2317448854446411 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5174053907394409 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22104665637016296 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6508564352989197 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.585834801197052 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5389963984489441 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9692605137825012 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9794080853462219 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8351702690124512 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5678589940071106 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02796127460896969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.758276641368866 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5733712315559387 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.994822084903717 /particle/accuracy_4_of_4‚0.9361499547958374 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5110383629798889 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8238232135772705 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5443434119224548 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7222027778625488 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5426396727561951 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6347001194953918 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21485069394111633 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2808782458305359 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6857898831367493 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5392701625823975 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9882935881614685 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.874312162399292 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5369049906730652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9932204484939575 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚98 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15575115382671356 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6962867379188538 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1300000101327896 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.11738027632236481 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05293876305222511 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15497736632823944 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9915350079536438 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21430881321430206 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.67212975025177 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6237150430679321 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 epoch‚2 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988176167011261 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518519401550293 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.41240909695625305 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20697146654129028 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9782658815383911 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9873934984207153 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5972272157669067 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_p†Ë‚–Ðroportion‚0.02457682229578495 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7451395988464355 jet/accuracy_5_of_5‚0 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.58915776014328 *jet/accuracy_1_of_1‚0.2971208989620209 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10490254312753677 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5103936195373535 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32285037636756897 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16657742857933044 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2136489748954773 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02972085401415825 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25171899795532227 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5122750997543335 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8973293304443359 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7validation_average_jet_accuracy‚0.33716750144958496 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913692474365234 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5443434119224548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268519401550293 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1670447140932083 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22139324247837067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23571941256523132 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12063024938106537 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8360228538513184 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5184099078178406 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.762377142906189 +jet/accuracy_2_of_3‚0.44315266609191895 /particle/accuracy_3_of_4‚0.9915520548820496 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5419201850891113 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1684546172618866 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2311001718044281 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9867972135543823 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3209577202796936 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6149389147758484 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16719038784503937 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21648436784744263 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9709391593933105 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 /particle/accuracy_2_of_2‚0.9998962879180908 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 *loss/val_total_loss‚11.743525066347544 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17547476291656494 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23172689974308014 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22139324247837067 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.648436963558197 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24727560579776764 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9896080493927002 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9752720594406128 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.23734726011753082 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12063024938106537 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 +jet/accuracy_3_of_3‚0.05962934345006943 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5640951991081238 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5132036209106445 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1807863861322403 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24304893612861633 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10291090607643127 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23058626055717468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4618176817893982 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7222027778625488 particle/accuracy_1_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7048598527908325ÁÓŠñÌ=ñÆ{ 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22104665637016296 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6508564352989197 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.585834801197052 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5389963984489441 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9692605137825012 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9794080853462219 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8351702690124512 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5678589940071106 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02796127460896969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.758276641368866 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5733712315559387 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.994822084903717 /particle/accuracy_4_of_4‚0.9361499547958374 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5110383629798889 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8238232135772705 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5443434119224548 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7222027778625488 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5426396727561951 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6347001194953918 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21485069394111633 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2808782458305359 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6857898831367493 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5392701625823975 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9882935881614685 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.874312162399292 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5369049906730652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9932204484939575 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚98 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15575115382671356 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6962867379188538 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1300000101327896 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.11738027632236481 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05293876305222511 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15497736632823944 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9915350079536438 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21430881321430206 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.67212975025177 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6237150430679321 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 epoch‚2 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988176167011261 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518519401550293 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.41240909695625305 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20697146654129028 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9782658815383911 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9873934984207153 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5972272157669067 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7451395988464355 jet/accuracy_5_of_5‚0 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.58915776014328 *jet/accuracy_1_of_1‚0.2971208989620209 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10490254312753677 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5103936195373535 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32285037636756897 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16657742857933044 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2136489748954773 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02972085401415825 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25171899795532227 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5122750997543335 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8973293304443359 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7validation_average_jet_accuracy‚0.33716750144958496 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913692474365234 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5443434119224548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268519401550293 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1670447140932083 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22139324247837067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23571941256523132 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12063024938106537 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8360228538513184 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5184099078178406 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.762377142906189 +jet/accuracy_2_of_3‚0.44315266609191895 /particle/accuracy_3_of_4‚0.9915520548820496 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5419201850891113 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1684546172618866 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2311001718044281 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9867972135543823 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3209577202796936 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6149389147758484 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16719038784503937 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21648436784744263 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9709391593933105 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 /particle/accuracy_2_of_2‚0.9998962879180908 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 *loss/val_total_loss‚11.743525066347544 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17547476291656494 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23172689974308014 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22139324247837067 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.648436963558197 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24727560579776764 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9896080493927002 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9752720594406128 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.23734726011753082 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12063024938106537 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 +jet/accuracy_3_of_3‚0.05962934345006943 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5640951991081238 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5132036209106445 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1807863861322403 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24304893612861633 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10291090607643127 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23058626055717468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4618176817893982 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7222027778625488 particle/accuracy_1_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7048598527908325 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8830059766769409 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6943564414978027 *jet/accuracy_1_of_3‚0.8944398760795593 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5832611322402954 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9021395444869995 + Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5458648800849915 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6326325535774231 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7499622106552124 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5683817863464355 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038793593645095825 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20365417003631592 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21937336027622223 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5902802348136902 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_3_of_4‚0.07392362505197525 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28774183988571167 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5603033900260925 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5651887655258179 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15438701212406158 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9856351017951965 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021938273683190346 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16992919147014618 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9940263628959656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23074877262115479 _runtime‚1041.467437962 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7426303625106812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9854729175567627 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929095506668091 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20398221909999847 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518519401550293 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929095506668091 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3710421919822693 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843413827729933e+09 *jet/accuracy_1_of_4‚0.9613498449325562 trainer/global_step‚1199 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9960276484489441 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04107537865638733 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7185032963752747 'particle/f_score‚0.6059921383857727 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.471800297498703 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9753967523574829 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5910869240760803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5808545351028442 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9537187218666077 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5290012359619141 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45841166377067566 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9827951788902283 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6780819892883301 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666567325592 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5836498737335205 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2317448854446411 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5174053907394409 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969bäºxå‘ò‚(:† ÇëÒ” gpu.0.gpu‚60.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì®d«·ó‚(:¬ ÇëÒ‹×Ö—# proc.memory.rssMB‚ 2716.36328125 network.sent‚ 1.805714e+06) disk./.usagePercent‚40.09567984623491 disk.sda.out‚ 22.4140625 disk.dm-1.out‚ 22.4140625 network.recv‚322299 disk.nvme0n1.in‚ 2540.453125 disk.sda3.in‚0 disk.rbd1.out‚2.75 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.579365730285645& disk.nvme0c0n1.out‚3814.4287109375 disk.sda2.out‚0! disk.dm-0.out‚3814.4287109375 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚597.4546051025391 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 2540.453125$ disk.nvme0n1.out‚3814.4287109375 disk.dm-1.in‚0$ memory_percent‚95.71616649627686 disk.rbd1.in‚0( proc.memory.availableMB‚ 701.86328125 cpu‚26.505726933068424! disk.nvme0c0n1.in‚ 2540.453125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 22.4140625 proc.cpu.threads‚17â×]¤“ô‚(:ˆ žÇëÒ¹‹Î• gpu.0.gpu‚43.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚240.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ž‹D»õ‚(:° žÇëÒ«“Í— disk.sda1.out‚0) disk./.usagePercent‚40.09583293874039 disk.sda3.out‚ 23.109375 disk.rbd0.out‚0 network.sent‚ 1.862169e+06 network.recv‚326265 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 23.109375! disk.nvme0n1.in‚ 2541.41796875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.8125 disk.rbd0.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚597.4568862915039& disk.nvme0c0n1.out‚4073.4541015625* proc.memory.percent‚16.579365730285645 disk.dm-1.in‚0 disk.dm-1.out‚ 23.109375# disk.nvme0c0n1.in‚ 2541.41796875$ memory_percent‚97.21732139587402# proc.memory.rssMB‚ 2716.36328125 cpu‚26.498795032476828' proc.memory.availableMB‚ 455.9140625! disk.dm-0.out‚4073.4541015625 disk.sda.in‚0$ disk.nvme0n1.out‚4073.4541015625 disk.dm-0.in‚ 2541.41796875 disk.sda1.in‚0Ù€ßRœö‚(:‘ ­ÇëÒ…й“ gpu.0.gpu‚26.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Öж÷‚(:« ­ÇëÒƒ­í– disk.sda1.out‚0 disk.dm-0.in‚ 2541.8125 disk.sda.in‚0( proc.memory.availableMB‚ 456.36328125 disk.sda2.out‚0 network.sent‚ 1.867195e+06 disk.rbd1.out‚2.8125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2715.7734375 cpu‚26.48825186230201 disk.sda3.out‚ 23.65234375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 2541.8125% disk.nvme0n1.out‚4075.93994140625 disk.sda3.in‚0" disk.dm-0.out‚4075.93994140625 network.recv‚329852) disk./.usagePercent‚40.09585239531299 disk.rbd1.in‚0 disk.dm-1.out‚ 23.65234375' disk.nvme0c0n1.out‚4075.93994140625 disk.nvme0n1.in‚ 2541.8125$ memory_percent‚97.21457958221436 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 23.65234375) proc.memory.percent‚16.57576560974121 disk.rbd0.out‚0 disk.sda2.in‚0$ disk./.usageGB‚597.4571762084961ЃŠÖ2ø- _wandbruntime‚1083  _runtime‚1083éý¦Åù¿ _step‚99 & lr-AdamW/pg1‚0.0004470011047278734 & lr-AdamW/pg2‚0.0004470011047278734 trainer/global_step‚1249 $ _timestamp‚1.784341424401672e+09 _runtime‚1083.096369683“Ë CÉúà trainer/global_step‚1249 $ _timestamp‚1.784341424401672e+09 _runtime‚1083.096369683 _step‚99 & lr-AdamW/pg1‚0.0004470011047278734 & lr-AdamW/pg2‚0.0004470011047278734c¤ôœ2û- _wandbruntime‚1083  _runtime‚1083S×›O/üÏO OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4890508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1319502538874 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1321178141696 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4890769 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1337517365760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188899596259328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11579033 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11569056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsM7uµ¦onitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2695820 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11569056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2695820 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191345242733056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11588099 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2697825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 _runtime‚1083.709921385 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843414250160973e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190023051964416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190023069003776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ž­áBL!€ÞM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190366279973906 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11588099 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2697825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 _runtime‚1083.709921385 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843414250160973e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190023051964416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190023069003776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4894539 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43329530265600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1338543255552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190366637563268 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191345242733056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceþ5@˜StatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 B2- _wandbruntime‚1083  _runtime‚1083ø‚€ _runtime‚1083.712345302 _step‚102 2loss/SRqqt2/assignment_loss‚0.6389964544635086 1loss/SRqqt2/detection_loss‚0.7478413077238374 .loss/FBt2/detection_loss‚0.669890014320143 epoch‚3 .loss/FRt2/detection_loss‚1.424307583681322 &loss/total_loss‚11.308290964406977 0loss/FRt2/assignment_loss‚1.2901667183773085 2loss/SRqqt1/assignment_loss‚0.8432555560669432 1loss/FBt1/assignment_loss‚0.35227714701848584 /loss/FRt1/detection_loss‚0.6703471566071265 1loss/SRqqt1/detection_loss‚1.2816197522772863 .loss/FBt1/detection_loss‚1.091076071584417 0loss/FRt1/assignment_loss‚2.2565898488898304 2loss/FBt2/assignment_loss‚0.041923353396768094 trainer/global_step‚1249 % _timestamp‚1.7843414250189576e+09š”h ƒ„ 2loss/SRqqt2/assignment_loss‚0.6389964544635086 1loss/SRqqt2/detection_loss‚0.7478413077238374 .loss/FBt2/detection_loss‚0.669890014320143 epoch‚3 .loss/FRt2/detection_loss‚1.424307583681322 &loss/total_loss‚11.308290964406977 0loss/FRt2/assignment_loss‚1.2901667183773085 2loss/SRqqt1/assignment_loss‚0.8432555560669432 1loss/FBt1/assignment_loss‚0.35227714701848584 /loss/FRt1/detection_loss‚0.6703471566071265 1loss/SRqqt1/detection_loss‚1.2816197522772863 .loss/FBt1/detection_loss‚1.091076071584417 0loss/FRt1/assignment_loss‚2.2565898488898304 2loss/FBt2/assignment_loss‚0.041923353396768094 trainer/global_step‚1249 % _timestamp‚1.7843414250189576e+09 _runtime‚1083.712345302 _step‚102f6}°wž„‚(:“ ¼ÇëҦ㺕 gpu.0.gpu‚54.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýlý¯…‚(:¤ ¼ÇëÒÄÅ– disk.rbd0.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚4089.62548828125$ memory_percent‚97.21782207489014 proc.cpu.threads‚17) disk./.usagePercent‚40.09563837301437 disk.dm-1.in‚0" disk.dm-0.out‚4089.62548828125 disk.sda.out‚ 24.08203125 disk.sda2.in‚0* proc.memory.percent‚16.579389572143555 network.sent‚ 1.872051e+06# disk./.usageGB‚597.453987121582 disk.sda3.in‚0 disk.drbd2165.out‚0 network.recv‚333111 disk.sda3.out‚ 24.08203125 disk.rbd1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚4089.62548828125 disk.rbd1.out‚2.875 disk.drbd2165.in‚0 disk.dm-0.in‚2542.25 disk.nvme0n1.in‚2542.25( proc.memory.availableMB‚ 455.83203125 disk.sda1.out‚0" proc.memory.rssMB‚ 2716.3671875 cpu‚26.47796697845536 disk.rbd0.out‚0 disk.dm-1.out‚ 24.08203125 disk.nvme0c0n1.in‚2542.25 disk.sda2.out‚07ÌôΓ†‚(:ˆ ËÇëÒë쾓 gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰#ÌÙ»‡‚(:° ËÇëÒºÈÕ–* disk./.usagePercent‚40.095636068946554 disk.rbd1.out‚2.875 disk.dm-1.in‚0 disk.dm-0.in‚ 2693.2265625" proc.memory.rssMB‚ 2715.7734375) proc.memory.percent‚16.57576560974121 proc.cpu.threads‚17$ disk./.usageGB‚597.4539527893066 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 2693.2265625$ memory_percent‚97.21536636352539 disk.sda.out‚ 24.35546875 disk.rbd0.in‚0 disk.sda.in‚0! disk.dm-0.out‚4098.5380859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚4098.5380859375 cpu‚26.467274942907313& proc.memory.availableMB‚ 456.234375 network.sent‚ 1.934893e+06 network.recv‚337645 disk.sda3.out‚ 24.35546875 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 2693.2265625$ disk.nvme0n1.out‚4098.5380859375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 24.35546875 disk.drbd2165.out‚02™ï2ˆ- _wandbruntime‚1122  _runtime‚1122pÄ„áƉÀ & lr-AdamW/pg2‚0.0004467498367173677 trainer/global_step‚1299 % _timestamp‚1.7843414641749754e+09 _runtime‚ 1122.86854367 _step‚103 & lr-AdamW/pg1‚0.0004467498367173677’snÊŠÄ % _timestamp‚1.7843414641749754e+09 _runtime‚ 1122.86854367 _step‚103 & lr-AdamW/pg1‚0.0004467498367173677 & lr-AdamW/pg2‚0.0004467498367173677 trainer/global_step‚1299gFîvX2‹- _wandbruntime‚1122  _runtime‚1122·[FØ'ŒÒO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2281941 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388744171008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372214982656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1122.869781297 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198610094438400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12023508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22310966 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2798887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 trainer/global_step‚1299 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45017845433856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 # _timestamp‚1.78434146417605e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45017738244608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197237862416384 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198610094438400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43629082650624<ì˼Ü'ÖO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45017738244608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197237862416384 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198610094438400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43629082650624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2281941 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388744171008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372214982656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1122.869781297 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198610094438400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12023508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22310966 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2798887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 trainer/global_step‚1299 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45017845433856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 # _timestamp‚1.78434146417605e+09h |½•2Ž- _wandbruntime‚1123  _runtime‚1123’ÎåAÞM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚1123.484057654 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43662530021376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197385120485376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45052186846208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843414647902458e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12032560 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197742787643776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0_¥fBè&âM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1299 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚1123.484057654 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43662530021376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197385120485376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45052186846208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843414647902458e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12032560 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197742787643776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090i’ ¶2‘- _wandbruntime‚1123  _runtime‚1123SSã?’ÿ &loss/total_loss‚10.425008697152322 epoch‚3 _step‚106 .loss/FRt2/detection_loss‚1.389185329316404 1loss/SRqqt2/detection_loss‚0.7726783764410726 /loss/FBt1/detection_loss‚0.7868647536432202 /loss/FRt1/assignment_loss‚2.209282672174828 /loss/FRt2/assignment_loss‚1.273613924042999 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚0.7005771439523925 1loss/FBt1/assignment_loss‚0.19001250711847784 /loss/FRt1/detection_loss‚0.6435248265455032 1loss/SRqqt1/detection_loss‚1.0826177470181593 % _timestamp‚1.7843414647928863e+09 _runtime‚1123.486186694 2loss/SRqqt2/assignment_loss‚0.6864369285039649 1loss/FBt2/assignment_loss‚0.05568876951634513 /loss/FBt2/detection_loss‚0.6345257188789547Ôä’> “ƒ trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚0.7005771439523925 1loss/FBt1/assignment_loss‚0.19001250711847784 /loss/FRt1/detection_loss‚0.6435248265455032 1loss/SRqqt1/detection_loss‚1.0826177470181593 % _timestamp‚1.7843414647928863e+09 _runtime‚1123.486186694 2loss/SRqqt2/assignment_loss‚0.6864369285039649 1loss/FBt2/assignment_loss‚0.05568876951634513 /loss/FBt2/detection_loss‚0.6345257188789547 &loss/total_loss‚10.425008697152322 epoch‚3 _step‚106 .loss/FRt2/detection_loss‚1.389185329316404 1loss/SRqqt2/detection_loss‚0.7726783764410726 /loss/FBt1/detection_loss‚0.7868647536432202 /loss/FRt1/assignment_loss‚2.209282672174828 /loss/FRt2/assignment_loss‚1.273613924042999jêsÈVž”‚(:“ ÚÇëÒàÿ†” gpu.0.gpu‚90.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.875' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.775000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎF³•‚(:¨ ÚÇëÒô–Æ—! disk.nvme0c0n1.in‚ 2698.390625! disk.dm-0.out‚4105.8798828125 disk.nvme0n1.in‚ 2698.390625 disk.dm-1.out‚ 24.66796875# memory_percent‚97.2198486328125& disk.nvme0c0n1.out‚4105.8798828125 disk.sda2.out‚0! proc.memory.availableMB‚455.5 disk.sda.in‚0 disk.sda3.in‚0* proc.memory.percent‚16.579389572143555 cpu‚26.457690581260962 network.sent‚ 1.952954e+06 disk.rbd1.out‚2.9375 disk.rbd0.in‚0 disk.dm-0.in‚ 2698.390625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2716.3671875 proc.cpu.threads‚17) disk./.usagePercent‚40.09563069278833 disk.drbd2165.in‚0$ disk.nvme0n1.out‚4105.8798828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚340911$ disk./.usageGB‚597.4538726806641 disk.drbd2165.out‚0 disk.sda.out‚ 24.66796875 disk.sda3.out‚ 24.66796875 disk.sda2.in‚0 disk.rbd1.in‚0î º–“–‚(:ˆ éÇëÒá…ç“ gpu.0.gpu‚88.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Af}!¸—‚(:­ éÇëÒèâΖ disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 2699.22265625) proc.memory.percent‚16.57576560974121 cpu‚26.448437849052933$ disk./.usageGB‚597.4539947509766 disk.rbd0.in‚0 disk.dm-1.out‚ 24.859375# disk.nvme0c0n1.in‚ 2699.22265625 disk.sda.out‚ 24.859375 disk.rbd1.in‚0 disk.rbd1.out‚2.9375& proc.memory.availableMB‚ 456.515625$ disk.nvme0n1.out‚4113.0087890625" proc.memory.rssMB‚ 2715.7734375) disk./.usagePercent‚40.09563888502943& disk.nvme0c0n1.out‚4113.0087890625! disk.dm-0.out‚4113.0087890625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 2699.22265625 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 2.002767e+06 disk.drbd2165.out‚0 network.recv‚345222 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 24.859375$ memory_percent‚97.21364974975586 disk.rbd0.out‚0 ÌeÏž˜‚(:“ øÇëÒÛ »“ gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uL ²™‚(:§ øÇëÒ”Ї– disk.rbd1.out‚3# disk./.usageGB‚598.249397277832 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 cpu‚26.439190731375422 proc.cpu.threads‚17' disk.nvme0c0n1.out‚4938.35498046875 disk.nvme0n1.in‚ 2700.1875* proc.memory.percent‚16.579413414001465 disk.drbd2165.out‚0" disk.dm-0.out‚4938.35498046875 disk.sda2.in‚0( proc.memory.availableMB‚ 455.76953125# proc.memory.rssMB‚ 2716.37109375) disk./.usagePercent‚40.14901901599387 disk.rbd0.in‚0 disk.sda.out‚ 25.08984375 network.sent‚ 2.007734e+06 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 2700.1875# memory_percent‚97.2182035446167 disk.sda3.out‚ 25.08984375 disk.rbd0.out‚0% disk.nvme0n1.out‚4938.35498046875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 25.08984375 disk.dm-0.in‚ 2700.1875 network.recv‚348438—Ü92š- _wandbruntime‚1162  _runtime‚1162!ä´ÕÆ›À _step‚107 & lr-AdamW/pg1‚0.0004464842964860989 & lr-AdamW/pg2‚0.0004464842964860989 trainer/global_step‚1349 $ _timestamp‚1.784341504173464e+09 _runtime‚1162.867074981ëï7©ÊœÄ _step‚107 & lr-AdamW/pg1‚0.0004464842964860989 & lr-AdamW/pg2‚0.0004464842964860989 trainer/global_step‚1349 $ _timestamp‚1.784341504173464e+09 _runtime‚1162.867074981kln#2- _wandbruntime‚1162  _runtime‚1162­1ËŒÙ'žÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46740408002048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204971058409328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204971415998578 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206022841421824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1422569145344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1422910484480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45300539189248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/êå¨pØrequested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1422910484480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45300539189248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019lø€ó=2 - _wandbruntime‚1163  _runtime‚1163 í^eã&¡ÝM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206171095085568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204747154927616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46774853236736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676692 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2370936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784341504790018e+09 _step‚109 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205118580134922 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2370933 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1421746566524 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440992471040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440903901184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1423923118592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423923118592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1423581772288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45333967947776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1163.483740518 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676690 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204747171966976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676692 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2370936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784341504790018e+09 _step‚109 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205118580134922 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2370933 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1421746566524 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440992471040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440903901184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1423923118592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423923118592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1423581772288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45333967947776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1163.483740518 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676690 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204747171966976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843415440114417e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚1399 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1473605986304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1473264647168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚1202.705135701 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660ðÁ+åÞ'±ØO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚1202.705135701 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213435230026752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843415440114417e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚1399 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1473605986304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1473264647168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0p»µÔû2²- _wandbruntime‚1203  _runtime‚1203Ô¡Uä&³ÞM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211022707712000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _runtime‚1203.320024943 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212109223448576 _step‚113 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5465755 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47005387261952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843415446262739e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_endOsC9#/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5465755 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47005387261952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843415446262739e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211022707712000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569q@t¾á2µ- _wandbruntime‚1203  _runtime‚1203 ò6<¶ÿ 2loss/SRqqt2/assignment_loss‚0.6568668182196816 /loss/FRt1/detection_loss‚0.6686524257966979 &loss/total_loss‚11.236104028804306 2loss/SRqqt1/assignment_loss‚0.8734387006987092 /loss/FRt2/detection_loss‚1.4722528308595964 0loss/SRqqt1/detection_loss‚1.057113110474855 /loss/FBt1/detection_loss‚1.5721775149197683 trainer/global_step‚1399 _step‚114 /loss/FRt1/assignment_loss‚2.238261650698107 0loss/FRt2/assignment_loss‚1.1159590241306758 1loss/FBt1/assignment_loss‚0.15551813224231845 2loss/FBt2/assignment_loss‚0.049014042419922056 1loss/SRqqt2/detection_loss‚0.7525612532548355 epoch‚3 $ _timestamp‚1.784341544629057e+09 _runtime‚ 1203.32235907 /loss/FBt2/detection_loss‚0.6242885250891387™ç.I ·ƒ /loss/FBt2/detection_loss‚0.6242885250891387 2loss/SRqqt2/assignment_loss‚0.6568668182196816 /loss/FRt1/detection_loss‚0.6686524257966979 &loss/total_loss‚11.236104028804306 2loss/SRqqt1/assignment_loss‚0.8734387006987092 /loss/FRt2/detection_loss‚1.4722528308595964 0loss/SRqqt1/detection_loss‚1.057113110474855 /loss/FBt1/detection_loss‚1.5721775149197683 trainer/global_step‚1399 _step‚114 /loss/FRt1/assignment_loss‚2.238261650698107 0loss/FRt2/assignment_loss‚1.1159590241306758 1loss/FBt1/assignment_loss‚0.15551813224231845 2loss/FBt2/assignment_loss‚0.049014042419922056 1loss/SRqqt2/detection_loss‚0.7525612532548355 epoch‚3 $ _timestamp‚1.784341544629057e+09 _runtime‚ 1203.32235907r¾†¬ž¸‚(:“ ´Èëқؤ“ gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:yµdµ¹‚(:ª ´ÈëÒÉà™•$ memory_percent‚97.22144603729248 proc.memory.rssMB‚2716.375 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda.out‚ 26.29296875! disk.nvme0n1.in‚ 2845.13671875* proc.memory.percent‚16.579437255859375 cpu‚26.40397137604886 network.sent‚ 2.085478e+06 disk.sda3.in‚0 disk.dm-1.out‚ 26.29296875" disk.nvme0n1.out‚ 4966.87109375 disk.rbd1.out‚3.125 disk.rbd0.out‚0 disk.dm-0.in‚ 2845.13671875( proc.memory.availableMB‚ 455.23828125 disk.sda2.out‚0$ disk./.usageGB‚597.4831199645996 disk.sda3.out‚ 26.29296875 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚362617 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 4966.87109375 disk.dm-0.out‚ 4966.87109375 proc.cpu.threads‚17) disk./.usagePercent‚40.09759350255336 disk.sda1.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2845.13671875®é;j“º‚(:ˆ ÃÈëÒå檓 gpu.0.gpu‚70.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”¦´»‚(:© ÃÈëÒ£Ò®– disk.dm-1.in‚0 disk.sda.out‚ 26.5390625 disk.sda2.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚4975.24560546875 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚26.395928884398632 disk.dm-1.out‚ 26.5390625 disk.nvme0c0n1.in‚ 2845.96875 disk.sda3.out‚ 26.5390625 disk.dm-0.in‚ 2845.96875% disk.nvme0n1.out‚4975.24560546875 disk.rbd1.out‚3.125 disk.sda3.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚4975.24560546875$ memory_percent‚97.21572399139404( proc.memory.availableMB‚ 456.17578125) proc.memory.percent‚16.57581329345703 network.sent‚ 2.148534e+06 disk.sda.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚40.09759427057596 disk.rbd1.in‚0 proc.memory.rssMB‚ 2715.78125 disk.drbd2165.in‚0 network.recv‚367449$ disk./.usageGB‚597.4831314086914 disk.nvme0n1.in‚ 2845.96875"üô«2¼- _wandbruntime‚1242  _runtime‚1242T'ɽà trainer/global_step‚1449 % _timestamp‚1.7843415840435083e+09 _runtime‚1242.737083115 _step‚115 ' lr-AdamW/pg1‚0.00044591046849746597 ' lr-AdamW/pg2‚0.00044591046849746597ŽcÍ¾Ç ' lr-AdamW/pg1‚0.00044591046849746597 ' lr-AdamW/pg2‚0.00044591046849746597 trainer/global_step‚1449 % _timestamp‚1.7843415840435083e+09 _runtime‚1242.737083115 _step‚115sV‹má2¿- _wandbruntime‚1242  _runtime‚1242oO/Ú'ÀÔO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220847977010176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚116 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3108316 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1521968486506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13396841 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 trainer/global_step‚1449 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1242.738669119 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 trainer/global_step‚1449 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1242.738669119 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.pea»Š:r'k‚539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3110334 _runtime‚1243.352942583 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439892 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13395916 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50220245941248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48676843800576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220996230666752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1524972775936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843415846591504e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50220142953472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439890 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1525314122240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚117 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219471257890816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13405907 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1524972775936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843415846591504e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50220142953472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439890 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1525314122240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚117 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219471257890816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13405907 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3110334 _runtime‚1243.352942583 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439892 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13395916 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50220245941248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48676843800576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220996230666752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0u¬öê2Å- _wandbruntime‚1243  _runtime‚1243-Ë~AÆü epoch‚3 trainer/global_step‚1449 /loss/FBt1/detection_loss‚1.8511542766354978 _runtime‚1243.355893033 _step‚118 2loss/SRqqt1/assignment_loss‚0.7845185896824454 .loss/FRt1/detection_loss‚0.667047718924504 1loss/SRqqt2/detection_loss‚0.7355797433587856 /loss/FBt2/detection_loss‚0.5168851367394941 $ _timestamp‚1.784341584662607e+09 /loss/FRt2/detection_loss‚1.4163007522617048 &loss/total_loss‚11.570628021121799 /loss/FRt1/assignment_loss‚2.113333660856647 /loss/FRt2/assignment_loss‚1.344106518675428 2loss/SRqqt2/assignment_loss‚0.6480771312883186 1loss/FBt1/assignment_loss‚0.11945134344303672 /loss/FBt2/assignment_loss‚0.148073445920768 1loss/SRqqt1/detection_loss‚1.2260997033351693eßxŠÇ€ trainer/global_step‚1449 /loss/FBt1/detection_loss‚1.8511542766354978 _runtime‚1243.355893033 _step‚118 2loss/SRqqt1/assignment_loss‚0.7845185896824454 .loss/FRt1/detection_loss‚0.667047718924504 1loss/SRqqt2/detection_loss‚0.7355797433587856 /loss/FBt2/detection_loss‚0.5168851367394941 $ _timestamp‚1.784341584662607e+09 /loss/FRt2/detection_loss‚1.4163007522617048 &loss/total_loss‚11.570628021121799 /loss/FRt1/assignment_loss‚2.113333660856647 /loss/FRt2/assignment_loss‚1.344106518675428 2loss/SRqqt2/assignment_loss‚0.6480771312883186 1loss/FBt1/assignment_loss‚0.11945134344303672 /loss/FBt2/assignment_loss‚0.148073445920768 1loss/SRqqt1/detection_loss‚1.2260997033351693 epoch‚3v×ùsúžÈ‚(:“ ÒÈëÒ˨ü“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á"-¼É‚(:± ÒÈëÒùäø—$ disk.nvme0n1.out‚4979.5185546875! disk.dm-0.out‚4979.5185546875( proc.memory.availableMB‚ 455.44140625 disk.sda1.in‚0 disk.dm-0.in‚ 3021.80859375$ memory_percent‚97.22020626068115 disk.rbd0.out‚0 disk.sda.out‚ 26.82421875 disk.sda2.in‚0 disk.rbd1.out‚3.1875 network.sent‚ 2.166649e+06& disk.nvme0c0n1.out‚4979.5185546875 disk.rbd1.in‚0 proc.memory.rssMB‚2716.375 disk.sda.in‚0 disk.sda3.out‚ 26.82421875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0) disk./.usagePercent‚40.09759196650815 disk.drbd2165.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.579437255859375 cpu‚26.387664978353616 network.recv‚371024# disk./.usageGB‚597.483097076416 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 26.82421875# disk.nvme0c0n1.in‚ 3021.80859375! disk.nvme0n1.in‚ 3021.80859375¡ÍÝ“Ê‚(:ˆ áÈëÒїΓ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m螻˂(:° áÈëÒçÕ disk.sda3.out‚ 27.0859375 disk.rbd1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 456.05078125 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 3022.7734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0) proc.memory.percent‚16.57581329345703 network.recv‚375489 disk.sda1.in‚0 disk.dm-1.out‚ 27.0859375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚4987.74951171875" disk.dm-0.out‚4987.74951171875 disk.sda.in‚0 disk.nvme0n1.in‚ 3022.7734375 disk.rbd0.out‚0 disk.dm-0.in‚ 3022.7734375 disk.rbd0.in‚0$ memory_percent‚97.21648693084717 disk.drbd2165.out‚0% disk.nvme0n1.out‚4987.74951171875 disk.sda3.in‚0 disk.rbd1.out‚3.1875 cpu‚26.380152228598682) disk./.usagePercent‚40.09763446375884$ disk./.usageGB‚597.4837303161621 disk.sda.out‚ 27.0859375 proc.memory.rssMB‚ 2715.78125 network.sent‚ 2.216413e+06ß'YžÌ‚(:“ ðÈëÒΗړ gpu.0.gpu‚61.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ;Ÿ¸Í‚(:­ ðÈëÒ𕧖 proc.cpu.threads‚17( proc.memory.availableMB‚ 455.29296875 disk.sda3.out‚ 27.4140625 disk.sda1.out‚0 disk.dm-1.in‚0! disk.dm-0.out‚4997.7666015625$ memory_percent‚97.22111225128174 network.recv‚378960 disk.sda3.in‚0 disk.dm-1.out‚ 27.4140625 disk.drbd2165.in‚0 disk.sda2.in‚0$ disk./.usageGB‚597.4836273193359 disk.rbd0.in‚0) disk./.usagePercent‚40.09762755155541 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2716.375 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3023.60546875 network.sent‚ 2.221391e+06 disk.rbd1.in‚0& disk.nvme0c0n1.out‚4997.7666015625* proc.memory.percent‚16.579437255859375 disk.rbd1.out‚3.25 disk.dm-0.in‚ 3023.60546875 disk.sda.out‚ 27.4140625! disk.nvme0n1.in‚ 3023.60546875$ disk.nvme0n1.out‚4997.7666015625 cpu‚26.372485597106973uÿVˆ2Î- _wandbruntime‚1282  _runtime‚1282™ÆÌKÈÏ _step‚119 ' lr-AdamW/pg1‚0.00044560221759600135 ' lr-AdamW/pg2‚0.00044560221759600135 trainer/global_step‚1499 $ _timestamp‚1.784341624186938e+09 _runtime‚1282.880585015:³EWÌÐÆ _step‚119 ' lr-AdamW/pg1‚0.00044560221759600135 ' lr-AdamW/pg2‚0.00044560221759600135 trainer/global_step‚1499 $ _timestamp‚1.784341624186938e+09 _runtime‚1282.880585015wíd«2Ñ- _wandbruntime‚1282  _runtime‚1282Gk›-ÒÔO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51908263337984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11813858 trainer/global_step‚1499 % _timestamp‚1.7843416241881292e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50314834356224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5842734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1282.881843972 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11813860 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13851293 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1572244423524 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 YIDeviceStatsMo•N˜F­nitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226686085382144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51908378915840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227099866240102 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1593428981760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0<ï…Þ'ÓØO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1574996989952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1574996989952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51908378915840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227099866240102 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1593428981760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51908263337984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11813858 trainer/global_step‚1499 % _timestamp‚1.7843416241881292e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50314834356224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5842734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1282.881843972 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11813860 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13851293 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1572244423524 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226686085382144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1283.497467193 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1576009624064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1575668277760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5846554 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226833309372416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11821492 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227247387965808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2632683 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚121 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7843416248036096e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3213599 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2632686 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088Ê/^õ<ÖâM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228408977650176 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51942824903168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50348263114752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11821490 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1283.497467193 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1576009624064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1575668277760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5846554 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226833309372416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11821492 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99q)[¬ 055616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227247387965808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2632683 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚121 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7843416248036096e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3213599 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2632686 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0yÕj6,2×- _wandbruntime‚1283  _runtime‚1283z+×êØ€ 0loss/FRt1/assignment_loss‚2.0870709154411187 /loss/FRt2/detection_loss‚1.2893655712919438 /loss/FBt2/detection_loss‚0.5428873952707721 _step‚122 epoch‚3 0loss/FRt2/assignment_loss‚1.2125874815033932 %loss/total_loss‚9.987403492059675 _runtime‚1283.499386175 0loss/FBt2/assignment_loss‚0.0729929436773762 /loss/FBt1/detection_loss‚0.7430412341907088 trainer/global_step‚1499 2loss/SRqqt1/assignment_loss‚0.7454095025716853 2loss/SRqqt2/assignment_loss‚0.6164578331763753 1loss/FBt1/assignment_loss‚0.15357529970190495 /loss/FRt1/detection_loss‚0.6402152709790749 1loss/SRqqt1/detection_loss‚1.1646475946855923 1loss/SRqqt2/detection_loss‚0.7191524495697296 % _timestamp‚1.7843416248060684e+09±ar Ù„ _step‚122 epoch‚3 0loss/FRt2/assignment_loss‚1.2125874815033932 %loss/total_loss‚9.987403492059675 _runtime‚1283.499386175 0loss/FBt2/assignment_loss‚0.0729929436773762 /loss/FBt1/detection_loss‚0.7430412341907088 trainer/global_step‚1499 2loss/SRqqt1/assignment_loss‚0.7454095025716853 2loss/SRqqt2/assignment_loss‚0.6164578331763753 1loss/FBt1/assignment_loss‚0.15357529970190495 /loss/FRt1/detection_loss‚0.6402152709790749 1loss/SRqqt1/detection_loss‚1.1646475946855923 1loss/SRqqt2/detection_loss‚0.7191524495697296 % _timestamp‚1.7843416248060684e+09 0loss/FRt1/assignment_loss‚2.0870709154411187 /loss/FRt2/detection_loss‚1.2893655712919438 /loss/FBt2/detection_loss‚0.5428873952707721z¹¸Ä‘Ú‚(:† ÿÈëÒ„´Š• gpu.0.gpu‚40.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¥Á½Û‚(:² ÿÈëÒ¼¥ì• disk.sda2.out‚0$ memory_percent‚97.21627235412598 disk.sda3.out‚ 27.66796875 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 27.66796875 disk.drbd2165.in‚0 disk.sda.in‚0 proc.memory.rssMB‚ 2715.78125 cpu‚26.364855764555617 network.sent‚ 2.22632e+06 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 3023.60546875 disk.dm-0.in‚ 3023.60546875 disk.rbd1.out‚3.25 disk.sda1.in‚0" disk.dm-0.out‚5000.32470703125' proc.memory.availableMB‚ 456.0859375' disk.nvme0c0n1.out‚5000.32470703125 disk.sda2.in‚0) proc.memory.percent‚16.57581329345703) disk./.usagePercent‚40.09762012733692$ disk./.usageGB‚597.4835166931152% disk.nvme0n1.out‚5000.32470703125 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚382501! disk.nvme0n1.in‚ 3023.60546875 disk.rbd0.in‚0 disk.sda.out‚ 27.66796875ûXݞ܂(:“ ŽÉëÒ¨ñ³” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ô}w¸Ý‚(:­ ŽÉëÒÓÁ‰—% disk.nvme0n1.out‚5011.37548828125 disk.sda3.out‚ 27.8515625 cpu‚26.357703037527298 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 3241.6171875" disk.dm-0.out‚5011.37548828125 disk.sda.in‚0 disk.dm-1.out‚ 27.8515625 network.sent‚ 2.289304e+06$ disk./.usageGB‚597.4835510253906' disk.nvme0c0n1.out‚5011.37548828125 disk.nvme0n1.in‚ 3241.6171875 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3241.6171875 disk.rbd0.out‚0 proc.memory.rssMB‚2716.375 disk.sda3.in‚0 disk.rbd1.out‚3.375 disk.rbd0.in‚0$ memory_percent‚97.22132682800293 proc.cpu.threads‚17 network.recv‚386724) disk./.usagePercent‚40.09762243140473 disk.sda2.in‚0' proc.memory.availableMB‚ 455.2578125* proc.memory.percent‚16.579437255859375 disk.sda1.out‚0 disk.sda.out‚ 27.8515625 disk.sda2.out‚0U¶œž‘Þ‚(:† ÉëÒ—ÇÒ” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í ö¹ß‚(:® ÉëÒÏÝÌ• disk.sda.out‚ 28.39453125 disk.dm-1.out‚ 28.39453125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 3242.44921875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚97.21856117248535 proc.memory.rssMB‚ 2715.78125 disk.drbd2165.in‚0) proc.memory.percent‚16.57581329345703* disk./.usagePercent‚40.097623711442395% disk.nvme0c0n1.out‚5019.345703125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 455.7109375 cpu‚26.350448195035188# disk./.usageGB‚597.483570098877 disk.dm-0.out‚5019.345703125 disk.rbd1.out‚3.375# disk.nvme0c0n1.in‚ 3242.44921875# disk.nvme0n1.out‚5019.345703125 disk.dm-0.in‚ 3242.44921875 disk.sda3.out‚ 28.39453125 disk.sda1.in‚0 network.sent‚ 2.294233e+06 network.recv‚390330 disk.rbd0.in‚0N,¾{2à- _wandbruntime‚1322  _runtime‚1322cõ4§ÇáÁ & lr-AdamW/pg1‚0.0004452797681832053 & lr-AdamW/pg2‚0.0004452797681832053 trainer/global_step‚1549 % _timestamp‚1.7843416641116228e+09 _runtime‚1322.805189708 _step‚123ƒR”ÆËâÅ & lr-AdamW/pg2‚0.0004452797681832053 trainer/global_step‚1549 % _timestamp‚1.7843416641116228e+09 _runtime‚1322.805189708 _step‚123 & lr-AdamW/pg1‚0.0004452797681832053{:['12ã- _wandbruntime‚1322  _runtime‚1322+j:\Ù'äÓO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚1549 _runtime‚1322.806455648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2718188 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚124 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6033839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6033571 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3315648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12195460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14295768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53630858041856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234475658731360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784341664112745e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3315648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12195460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14295768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53630858041856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234475658731360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784341664112745e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2718188 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚124|vÉìü2æ- _wandbruntime‚1323  _runtime‚1323–Òû/È çßM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6037626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52019701041152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 D;DeviceStatsMoIÆbdnitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12203092 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1623871964154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235821724633600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234195377893376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1626363779584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1323.420655036 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232999325532160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843416647269325e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1626705125888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234195377893376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1626363779584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1323.420655036 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232999325532160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843416647269325e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1626705125888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52019701041152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438}“eaG2é- _wandbruntime‚1323  _runtime‚1323Gô¹êÿ 1loss/SRqqt2/assignment_loss‚0.582561493184255 &loss/total_loss‚11.265450126422827 /loss/FRt1/assignment_loss‚2.193274839879563 /loss/FRt2/detection_loss‚1.3800171423754575 epoch‚3 1loss/FBt1/assignment_loss‚0.12876209237490338 /loss/FRt1/detection_loss‚0.6237133940986019 1loss/SRqqt1/detection_loss‚1.0829711170854415 /loss/FBt2/detection_loss‚0.6507741154556901 trainer/global_step‚1549 _step‚126 2loss/SRqqt1/assignment_loss‚0.7102355119514082 1loss/FBt2/assignment_loss‚0.07525418373206705 1loss/SRqqt2/detection_loss‚0.6801547025983442 /loss/FBt1/detection_loss‚1.9113467184728699 % _timestamp‚1.7843416647300289e+09 _runtime‚1323.423319586 /loss/FRt2/assignment_loss‚1.246384815214227¼)  ëƒ 1loss/FBt1/assignment_loss‚0.12876209237490338 /loss/FRt1/detection_loss‚0.6237133940986019 1loss/SRqqt1/detection_loss‚1.0829711170854415 /loss/FBt2/detection_loss‚0.6507741154556901 trainer/global_step‚1549 _step‚126 2loss/SRqqt1/assignment_loss‚0.7102355119514082 1loss/FBt2/assignment_loss‚0.07525418373206705 1loss/SRqqt2/detection_loss‚0.6801547025983442 /loss/FBt1/detection_loss‚1.9113467184728699 % _timestamp‚1.7843416647300289e+09 _runtime‚1323.423319586 /loss/FRt2/assignment_loss‚1.246384815214227 1loss/SRqqt2/assignment_loss‚0.582561493184255 &loss/total_loss‚11.265450126422827 /loss/FRt1/assignment_loss‚2.193274839879563 /loss/FRt2/detection_loss‚1.3800171423754575 epoch‚3~îøTžì‚(:“ ¬ÉëÒ¾Þì“ gpu.0.gpu‚89.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôë»í‚(:° ¬ÉëÒ¦œ¶–$ disk./.usageGB‚597.4837265014648 disk.rbd1.in‚0) proc.memory.percent‚16.57581329345703 disk.sda2.in‚0$ disk.nvme0n1.out‚5022.9443359375 disk.sda3.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 455.71484375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 3242.44921875 disk.dm-0.in‚ 3242.44921875 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2715.78125 network.sent‚ 2.312146e+06 disk.drbd2165.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 3242.44921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 28.6015625$ memory_percent‚97.21853733062744 disk.drbd2165.out‚0 disk.sda.out‚ 28.6015625 disk.rbd1.out‚3.4375 cpu‚26.34362738516675 disk.sda1.out‚0* disk./.usagePercent‚40.097634207751305 disk.dm-1.out‚ 28.6015625& disk.nvme0c0n1.out‚5022.9443359375 network.recv‚393828! disk.dm-0.out‚5022.9443359375aÂ{žî‚(:“ »ÉëÒ€Š±“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Œ65»ï‚(:° »ÉëÒ–( proc.memory.availableMB‚ 455.63671875 network.sent‚ 2.362017e+06$ disk./.usageGB‚597.4837303161621 disk.nvme0n1.in‚ 3243.4140625 disk.rbd1.in‚0 disk.rbd1.out‚3.4375 disk.dm-1.out‚ 28.8984375 proc.cpu.threads‚17 disk.sda3.out‚ 28.8984375 disk.drbd2165.out‚0 disk.sda3.in‚0" disk.dm-0.out‚5033.46142578125 proc.memory.rssMB‚ 2715.78125) disk./.usagePercent‚40.09763446375884 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 3243.4140625) proc.memory.percent‚16.57581329345703 network.recv‚398359% disk.nvme0n1.out‚5033.46142578125 disk.dm-0.in‚ 3243.4140625 disk.sda.out‚ 28.8984375$ memory_percent‚97.21901416778564 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 cpu‚26.336444525006353' disk.nvme0c0n1.out‚5033.46142578125 disk.rbd0.out‚0Œ1”•2ð- _wandbruntime‚1362  _runtime‚1362¬­ÑBÅñ¿ % _timestamp‚1.7843417041299238e+09 _runtime‚1362.823513141 _step‚127 % lr-AdamW/pg1‚0.000444943140969401 % lr-AdamW/pg2‚0.000444943140969401 trainer/global_step‚1599’=ü~Éòà _step‚127 % lr-AdamW/pg1‚0.000444943140969401 % lr-AdamW/pg2‚0.000444943140969401 trainer/global_step‚1599 % _timestamp‚1.7843417041299238e+09 _runtime‚1362.823513141øF ß2ó- _wandbruntime‚1362  _runtime‚1362wÐ*Ú'ôÔO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 _runtime‚1362.824743593 _step‚128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243086576338944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12577058 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3419232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1676046654464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240178348453888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241410171305984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14750220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1695724240896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12577058 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2805441 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6224407 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843417041309938e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14750220 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241410171305984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1676387993600 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1676046654464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ƒ¥±™ÓõÙO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12577060 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1676046654464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 _runtime‚1362.824743593 _step‚128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243086576338944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12577058 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3419232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1676046654464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240178348453888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241410171305984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14750220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1695724240896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12577058 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2805441 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6224407 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843417041309938e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14750220 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241410171305984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1676387993600 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14759272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1674488451062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224¨-xoé&øãM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240324842086400 _runtime‚1363.443098915 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14759272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241557412335616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843417047491543e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241999688126824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14759272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1674488451062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169QxK©2ù- _wandbruntime‚1363  _runtime‚1363Á6ÄÛú€ &loss/total_loss‚10.307170486762194 0loss/FRt1/assignment_loss‚2.1126931807712506 2loss/SRqqt1/assignment_loss‚0.8140887567322352 /loss/FRt1/detection_loss‚0.6995972521379037 1loss/SRqqt2/detection_loss‚0.7108670696952758 2loss/SRqqt2/assignment_loss‚0.6077393666402197 1loss/FBt2/assignment_loss‚0.05240139266345117 /loss/FBt1/detection_loss‚1.0735078568259875 epoch‚3 /loss/FBt2/detection_loss‚0.5992778385723275 0loss/FRt2/assignment_loss‚1.1150476731204444 1loss/FBt1/assignment_loss‚0.08179748321708757 .loss/FRt2/detection_loss‚1.288449045672723 0loss/SRqqt1/detection_loss‚1.151703570713289 _runtime‚1363.444064437 _step‚130 trainer/global_step‚1599 % _timestamp‚1.7843417047507753e+09³uœN û… &loss/total_loss‚10.307170486762194 0loss/FRt1/assignment_loss‚2.1126931807712506 2loss/SRqqt1/assignment_loss‚0.8140887567322352 /loss/FRt1/detection_loss‚0.6995972521379037 1loss/SRqqt2/detection_loss‚0.7108670696952758 2loss/SRqqt2/assignment_loss‚0.6077393666402197 1loss/FBt2/assignment_loss‚0.05240139266345117 /loss/FBt1/detection_loss‚1.0735078568259875 epoch‚3 /loss/FBt2/detection_loss‚0.5992778385723275 0loss/FRt2/assignment_loss‚1.1150476731204444 1loss/FBt1/assignment_loss‚0.08179748321708757 .loss/FRt2/detection_loss‚1.288449045672723 0loss/SRqqt1/detection_loss‚1.151703570713289 _runtime‚1363.444064437 _step‚130 trainer/global_step‚1599 % _timestamp‚1.7843417047507753e+09‚;R` “ü‚(:ˆ ÊÉëÒžÖŒ” gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©1´¶ý‚(:« ÊÉëÒ麖 network.sent‚ 2.367e+06 disk.sda1.in‚0! disk.dm-0.out‚5041.1220703125 proc.cpu.threads‚17 network.recv‚401725 disk.sda2.in‚0$ disk.nvme0n1.out‚5041.1220703125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 29.18359375 disk.drbd2165.in‚0 disk.sda.in‚0) disk./.usagePercent‚40.09762038334445 disk.rbd0.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 3334.2109375 disk.dm-0.in‚ 3334.2109375$ disk./.usageGB‚597.4835205078125 disk.rbd1.in‚0 disk.rbd1.out‚3.5 disk.nvme0n1.in‚ 3334.2109375 disk.dm-1.in‚0 disk.sda.out‚ 29.18359375& proc.memory.availableMB‚ 453.140625 cpu‚26.334448767067418 disk.sda3.in‚0 disk.dm-1.out‚ 29.18359375& disk.nvme0c0n1.out‚5041.1220703125$ memory_percent‚97.23424911499023" proc.memory.rssMB‚ 2717.6953125 disk.drbd2165.out‚0* proc.memory.percent‚16.587495803833008xôï“þ‚(:ˆ ÙÉëÒËÙ‰• gpu.0.gpu‚82.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µq‹‰ÿ‚(:¬ ÙÉëÒ¶™ö• disk.drbd2165.in‚0* disk./.usagePercent‚40.097623455434864 disk.sda.in‚0 disk.rbd1.in‚0* pÅGý.roc.memory.percent‚16.603612899780273 cpu‚26.368872587303905 network.sent‚ 2.42977e+06$ disk./.usageGB‚597.4835662841797 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚5046.12841796875 disk.sda3.in‚0" proc.memory.rssMB‚ 2720.3359375" disk.dm-0.out‚5046.12841796875 disk.nvme0n1.in‚ 3334.2109375% proc.memory.availableMB‚ 449.84375 disk.rbd0.out‚0 disk.dm-1.out‚ 29.5390625 disk.drbd2165.out‚0 disk.sda.out‚ 29.5390625 disk.sda2.out‚0# memory_percent‚97.2543716430664 disk.rbd0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚405437' disk.nvme0c0n1.out‚5046.12841796875 disk.sda3.out‚ 29.5390625 disk.rbd1.out‚3.5" disk.nvme0c0n1.in‚ 3334.2109375 disk.dm-0.in‚ 3334.2109375 disk.sda2.in‚0?òt¼2€ - _wandbruntime‚1380  _runtime‚1380ܹµ= ¯{ epoch‚3 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5764867067337036 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936254620552063 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3244454860687256 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5926983952522278 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06570789963006973 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9895555377006531 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12646864354610443 *jet/accuracy_1_of_3‚0.9254202246665955 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3763085901737213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6454000473022461 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7972877025604248 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954361915588379 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0925326943397522 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6434578895568848 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.717082679271698 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06431417167186737 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7311304807662964 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1858774572610855 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6660094261169434 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900725483894348 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,jet/accuracy_3_of_3‚0.042144306004047394 jet/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941173791885376 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.761725902557373 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18089264631271362 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7843417218745112e+09 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *loss/val_total_loss‚10.080069538212213 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2506484389305115 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23463496565818787 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8198363780975342 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951848387718201 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39737266302108765 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954361915588379 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5478547215461731 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.798385500907898 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966332316398621 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11030843108892441 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06384876370429993 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.608468770980835 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚131 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7099859714508057 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.775030255317688 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6213387250900269 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992418646812439 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06646345555782318 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8042706251144409 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38490167260169983 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_3_of_4‚0.04299583286046982 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8601261973381042 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9205324053764343 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8286847472190857 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028831226751208305 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9861377477645874 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10827045142650604 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23280790448188782 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7023385763168335 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.054780442267656326 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8198363780975342 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.995876133441925 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24120981991291046 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.055958788841962814 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6734457612037659 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.42616981267929077 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19901788234710693 /particle/accuracy_4_of_4‚0.9525304436683655 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4848221242427826 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10499028861522675 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8231292963027954 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20288389921188354 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7672616243362427 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1837579607963562 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04246235638856888 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9646103382110596 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6716765761375427 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22432658076286316 +jet/accuracy_1_of_1‚0.34216055274009705 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2931758761405945 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6469638347625732 +particle/sensitivity‚0.6043791174888611 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4382687509059906 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8033802509307861 _runtime‚1380.568712821 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29890674352645874 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8996821641921997 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2099466621875763 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7076563239097595 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20317906141281128 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940197467803955 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1594085991382599 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6685654520988464 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0949999988079071 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7995324730873108 .particle/accuracy_3_of_4‚0.990646243095398 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6716765761375427 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7364436388015747 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6899493932723999 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02409413829445839 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6586763262748718 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19350536167621613 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7148097157478333 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2520883083343506 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.055958788841962814 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800387918949127 particle/accuracy_1_of_3‚1 'particle/accuracy‚0.836266815662384 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 particle/accuracy_3_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21079058945178986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6919690370559692 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19776548445224762 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24173811078071594 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1599 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5975567102432251 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18765974044799805 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04548758640885353 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5755208730697632 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24397781491279602 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7714611291885376 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19002237915992737 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8369235992431641 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24120981991291046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20581787824630737 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.49096375703811646 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5927138328552246 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 *jet/accuracy_2_of_4‚0.7392649054527283 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6629554033279419 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7653807401657104 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8338057398796082 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928463697433472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6531237363815308 'particle/f_score‚0.6131932139396667 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23931936919689178 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7123058438301086 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7623204588890076 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5199999809265137 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21815942227840424 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5408982038497925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03983669355511665 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8038130402565002 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941030144691467 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7171711921691895 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1177898496389389 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808©‡ò®º=‚ ´{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06384876370429993 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.608468770980835 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚131 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7099859714508057 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.775030255317688 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6213387250900269 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992418646812439 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06646345555782318 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8042706251144409 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38490167260169983 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_3_of_4‚0.04299583286046982 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8601261973381042 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9205324053764343 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8286847472190857 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028831226751208305 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9861377477645874 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10827045142650604 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23280790448188782 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7023385763168335 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.054780442267656326 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8198363780975342 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.995876133441925 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24120981991291046 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.055958788841962814 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6734457612037659 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.42616981267929077 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19901788234710693 /particle/accuracy_4_of_4‚0.9525304436683655 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4848221242427826 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10499028861522675 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8231292963027954 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20288389921188354 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7672616243362427 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1837579607963562 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04246235638856888 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9646103382110596 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6716765761375427 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22432658076286316 +jet/accuracy_1_of_1‚0.34216055274009705 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2931758761405945 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6469638347625732 +particle/sensitivity‚0.6043791174888611 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4382687509059906 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8033802509307861 _runtime‚1380.568712821 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29890674352645874 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8996821641921997 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2099466621875763 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7076563239097595 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20317906141281128 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940197467803955 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1594085991382599 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6685654520988464 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0949999988079071 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7995324730873108 .particle/accuracy_3_of_4‚0.990646243095398 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6716765761375427 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7364436388015747 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6899493932723999 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02409413829445839 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6586763262748718 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19350536167621613 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7148097157478333 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2520883083343506 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.055958788841962814 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800387918949127 particle/accuracy_1_of_3‚1 'particle/accuracy‚0.836266815662384 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 particle/accuracy_3_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21079058945178986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6919690370559692 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19776548445224762 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24173811078071594 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚1599 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5975567102432251 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18765974044799805 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04548758640885353 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5755208730697632 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24397781491279602 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7714611291885376 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19002237915992737 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8369235992431641 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24120981991291046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20581787824630737 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.49096375703811646 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5927138328552246 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 *jet/accuracy_2_of_4‚0.7392649054527283 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6629554033279419 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7653807401657104 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8338057398796082 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928463697433472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6531237363815308 'particle/f_score‚0.6131932139396667 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23931936919689178 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7123058438301086 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7623204588890076 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5199999809265137 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21815942227840424 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5408982038497925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03983669355511665 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8038130402565002 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941030144691467 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7171711921691895 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1177898496389389 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 epoch‚3 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5764867067337036 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936254620552063 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3244454860687256 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5926983952522278 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06570789963006973 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9895555377006531 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12646864354610443 *jet/accuracy_1_of_3‚0.9254202246665955 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3763085901737213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6454000473022461 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7972877025604248 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954361915588379 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0925326943397522 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6434578895568848 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.717082679271698 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06431417167186737 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7311304807662964 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1858774572610855 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6660094261169434 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900725483894348 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,jet/accuracy_3_of_3‚0.042144306004047394 jet/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941173791885376 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.761725902557373 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18089264631271362 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7843417218745112e+09 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *loss/val_total_loss‚10.080069538212213 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2506484389305115 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23463496565818787 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8198363780975342 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951848387718201 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39737266302108765 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954361915588379 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5478547215461731 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.798385500907898 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966332316398621 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11030843108892441 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1ƒX€<åƒ ‚(:“ èÉëÒô¼Ê“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚8Gåò22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ûÄ„ ‚(:¹ èÉëÒöáÅ– disk.sda3.in‚0* proc.memory.percent‚16.603612899780273 cpu‚26.36092370104752* disk./.usagePercent‚40.097759139428014 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚5299.85595703125 disk.rbd1.in‚0$ disk./.usageGB‚597.4855880737305 disk.dm-0.in‚ 3335.17578125 disk.sda.out‚ 29.91796875 network.sent‚ 2.433978e+06# disk.nvme0c0n1.in‚ 3335.17578125$ memory_percent‚95.46868801116943 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.out‚3.5625 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚5299.85595703125 disk.sda.in‚0! disk.nvme0n1.in‚ 3335.17578125( proc.memory.availableMB‚ 742.41015625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 29.91796875" proc.memory.rssMB‚ 2720.3359375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5299.85595703125 network.recv‚408724 disk.dm-1.out‚ 29.91796875s  “… ‚(:ˆ ÷ÉëÒ¥¸¢• gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†5Éü¹† ‚(:® ÷ÉëÒÄÏ•# proc.memory.availableMB‚742.375* proc.memory.percent‚16.603612899780273 proc.cpu.threads‚17$ disk./.usageGB‚597.4903717041016 disk.sda1.out‚0 disk.dm-1.out‚ 31.05078125 disk.dm-0.in‚ 3336.0078125 disk.sda.in‚0 disk.sda.out‚ 31.05078125 disk.nvme0n1.in‚ 3336.0078125$ disk.nvme0n1.out‚5562.7099609375 disk.rbd1.out‚3.5625 disk.drbd2165.in‚0 network.recv‚413548) disk./.usagePercent‚40.09808017287595 disk.sda3.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.3359375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚5562.7099609375! disk.dm-0.out‚5562.7099609375 disk.sda2.out‚0 cpu‚26.354242325738852 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 3336.0078125 disk.rbd0.in‚0 disk.dm-1.in‚0$ memory_percent‚95.46890258789062 network.sent‚ 2.491547e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 31.05078125 ì‘Ù2‡ - _wandbruntime‚1422  _runtime‚1422@Ž—´Éˆ à _runtime‚1422.171825561 _step‚132 ' lr-AdamW/pg1‚0.00044459235757552545 ' lr-AdamW/pg2‚0.00044459235757552545 trainer/global_step‚1649 % _timestamp‚1.7843417634782286e+09eçr'Ή È trainer/global_step‚1649 % _timestamp‚1.7843417634782286e+09 _runtime‚1422.171825561 _step‚132 ' lr-AdamW/pg1‚0.00044459235757552545 ' lr-AdamW/pg2‚0.00044459235757552545„¢ݧ2Š - _wandbruntime‚1422  _runtime‚1422Ê. ÖÚ'‹ ÔO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1765063108096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57110232629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744816703488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251154341831690 % _timestamp‚1.7843417634793766e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744475364352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249412528222208 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15284976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57110123343360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3553352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚1422.176345853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 Iß'Œ ÙO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079324 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚133 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2895208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250701806477312 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚250701789437952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1742154159372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2895211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55345078847488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251154699420940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744816703488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15274999 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1765063108096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57110232629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744816703488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251154341831690 % _timestamp‚1.7843417634793766e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744475364352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249412528222208 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15284976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57110123343360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3553352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚1422.176345853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843417640953474e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1766176249856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249559055933440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1743165213206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1742824663204 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250849047506944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57144683855872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251302221146646 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252594518458880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1766083485696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDe-Û}¾viceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843417640953474e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1766176249856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249559055933440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1743165213206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1742824663204 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250849047506944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57144683855872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251302221146646 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269†¦ñD[2 - _wandbruntime‚1422  _runtime‚1422•QÕ±‘ þ 1loss/FBt2/assignment_loss‚0.05031988924849019 /loss/FRt2/detection_loss‚1.3824012310437148 /loss/FBt1/detection_loss‚1.1582617683819048 0loss/FRt2/assignment_loss‚1.3566035928582787 1loss/SRqqt2/assignment_loss‚0.683205564565344 1loss/SRqqt1/detection_loss‚1.1465766936065567 epoch‚4 /loss/FRt1/assignment_loss‚2.057483265937214 1loss/FBt1/assignment_loss‚0.35403090045070307 1loss/SRqqt2/detection_loss‚0.7830358997654461 /loss/FBt2/detection_loss‚0.6986256452909768 trainer/global_step‚1649 _runtime‚1422.790684395 /loss/FRt1/detection_loss‚0.6894912598729358 $ _timestamp‚1.784341764097396e+09 _step‚135 &loss/total_loss‚11.072080848944005 1loss/SRqqt1/assignment_loss‚0.712045137922439ëSd ’ ƒ epoch‚4 /loss/FRt1/assignment_loss‚2.057483265937214 1loss/FBt1/assignment_loss‚0.35403090045070307 1loss/SRqqt2/detection_loss‚0.7830358997654461 /loss/FBt2/detection_loss‚0.6986256452909768 trainer/global_step‚1649 _runtime‚1422.790684395 /loss/FRt1/detection_loss‚0.6894912598729358 $ _timestamp‚1.784341764097396e+09 _step‚135 &loss/total_loss‚11.072080848944005 1loss/SRqqt1/assignment_loss‚0.712045137922439 1loss/FBt2/assignment_loss‚0.05031988924849019 /loss/FRt2/detection_loss‚1.3824012310437148 /loss/FBt1/detection_loss‚1.1582617683819048 0loss/FRt2/assignment_loss‚1.3566035928582787 1loss/SRqqt2/assignment_loss‚0.683205564565344 1loss/SRqqt1/detection_loss‚1.1465766936065567‡ÝEÔ“ ‚(:„ †ÊëÒÖœ®” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·)àY»” ‚(:° †ÊëÒ¥‘­– network.sent‚ 2.496424e+06 disk.dm-1.out‚ 32.296875 disk.sda.in‚0 cpu‚26.347979608430773% disk.nvme0n1.out‚5572.04638671875# memory_percent‚95.4646348953247 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3434.28515625 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 3434.28515625" proc.memory.rssMB‚ 2719.7734375! disk.nvme0n1.in‚ 3434.28515625 disk.sda3.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚40.09795831328965 disk.sda2.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 743.07421875) proc.memory.percent‚16.60017967224121 disk.sda.out‚ 32.296875 disk.sda2.in‚0 disk.sda3.out‚ 32.296875 disk.rbd1.out‚3.625 disk.drbd2165.in‚0" disk.dm-0.out‚5572.04638671875 network.recv‚416786$ disk./.usageGB‚597.4885559082031 disk.dm-1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚5572.04638671875Am©‡’• ‚(:‡ •ÊëÒ³ßÉ“ gpu.0.gpu‚18.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡¢_½®– ‚(:£ •ÊëÒûË– disk.drbd2165.in‚0 disk.rbd1.out‚3.625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.46773433685303* proc.memory.percent‚16.603612899780273 disk.sda.in‚0 disk.nvme0n1.in‚3435.25 network.sent‚ 2.559532e+06 disk.dm-1.in‚0 disk.dm-0.in‚3435.25" proc.memory.rssMB‚ 2720.3359375 disk.dm-1.out‚ 34.4453125" disk.dm-0.out‚5586.19970703125 disk.sda.out‚ 34.4453125* disk./.usagePercent‚40.098127022254715$ disk./.usageGB‚597.4910697937012 disk.drbd2165.out‚0% disk.nvme0n1.out‚5586.19970703125 proc.cpu.threads‚17 network.recv‚421593 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 34.4453125( proc.memory.availableMB‚ 742.56640625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚3435.25' disk.nvme0c0n1.out‚5586.19970703125 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚26.34152372183222,»f— ‚(:’ ¤ÊëÒƒñ“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.&ÕY¯˜ ‚(:¤ ¤ÊëÒ©÷†– disk.nvme0c0n1.in‚3435.25 disk.nvme0n1.in‚3435.25" proc.memory.rssMB‚ 2719.7734375) proc.memory.percent‚16.60017967224121 network.sent‚ 2.564486e+06 disk.dm-1.out‚ 35.015625 disk.sda3.out‚ 35.015625 disk.sda.out‚ 35.015625 disk.rbd1.out‚ 3.69140625$ memory_percent‚95.46411037445068 proc.cpu.threads‚17 network.recv‚424916 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0) disk./.usagePercent‚40.09803588357252 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚3435.25 disk.sda2.out‚0( proc.memory.availableMB‚ 743.16015625' disk.nvme0c0n1.out‚5593.90576171875 disk.sda2.in‚0$ disk./.usageGB‚597.4897117614746 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚26.334889146831287 disk.rbd1.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚5593.90576171875 disk.sda.in‚0% disk.nvme0n1.out‚5593.90576171875rn2™ - _wandbruntime‚1462  _runtime‚14626^ÅøÇš Á & lr-AdamW/pg1‚0.0004442274405317405 & lr-AdamW/pg2‚0.0004442274405317405 trainer/global_step‚1699 % _timestamp‚1.7843418036226792e+09 _runtime‚1462.316397434 _step‚136Âþޙ̛ Æ & lr-AdamW/pg2‚0.0004442274405317405 trainer/global_step‚1699 % _timestamp‚1.7843418036226792e+09 _runtime‚1462.316397434 _step‚136 & lr-AdamW/pg1‚0.0004442274405317405ˆˆëÅÜ2œ - _wandbruntime‚1462  _runtime‚1462pœÔ*Ø' ÒO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1795512205312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460924 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1795512205312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2982461 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚ 1462.31768822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29200354 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784341803623936e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258063840919552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58832803577344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 _step‚137 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076Ÿ\üüþž ×O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚ 1462.31768822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29200354 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784341803623936e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258063840919552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58832803577344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 _step‚137 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1795512205312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460924 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1795512205312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2982461 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0‰üœ\–2Ÿ - _wandbruntime‚1462  _runtime‚1462`C¶ä&  ÞM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1793441150112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258211081949184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15748494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258211098988544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258678371227154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843418042400715e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 Pé&¡ ãM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1462.933817025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258211098988544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258678371227154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843418042400715e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1793441150112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258211081949184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15748494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16193880 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_ˆ"fe batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845866368000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13842526 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843418435209033e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265906641992706 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58687936088064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16193880 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845866368000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13842526 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843418435209033e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265906641992706 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58687936088064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267420012425728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1847220341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚1502.829549865 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150470144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3071456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983eà?ƒ³ ãM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150470144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3071456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13850156 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1868675858432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267420012425728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1847220341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚1502.829549865 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Žš”¡!2´ - _wandbruntime‚1502  _runtime‚1502óEŵ ÿ 1loss/FBt1/assignment_loss‚0.08817449960301271 /loss/FRt1/detection_loss‚0.5867619436833621 /loss/FBt2/detection_loss‚0.6943929304307612 0loss/FRt2/assignment_loss‚1.1693921329139114 2loss/SRqqt2/assignment_loss‚0.5580620351172436 0loss/SRqqt1/detection_loss‚1.103190235572012 1loss/SRqqt2/detection_loss‚0.6829523800853586 /loss/FBt1/detection_loss‚0.9962026858702302 epoch‚4 _step‚143 2loss/SRqqt1/assignment_loss‚0.6220418052837862 1loss/FBt2/assignment_loss‚0.06275216287754705 /loss/FRt2/detection_loss‚1.2705303901125693 %loss/total_loss‚9.917889277437975 /loss/FRt1/assignment_loss‚2.083436075888182 trainer/global_step‚1749 % _timestamp‚1.7843418441377277e+09 _runtime‚1502.8310068058ì« ¶ „ /loss/FRt1/detection_loss‚0.5867619436833621 /loss/FBt2/detection_loss‚0.6943929304307612 0loss/FRt2/assignment_loss‚1.1693921329139114 2loss/SRqqt2/assignment_loss‚0.5580620351172436 0loss/SRqqt1/detection_loss‚1.103190235572012 1loss/SRqqt2/detection_loss‚0.6829523800853586 /loss/FBt1/detection_loss‚0.9962026858702302 epoch‚4 _step‚143 2loss/SRqqt1/assignment_loss‚0.6220418052837862 1loss/FBt2/assignment_loss‚0.06275216287754705 /loss/FRt2/detection_loss‚1.2705303901125693 %loss/total_loss‚9.917889277437975 /loss/FRt1/assignment_loss‚2.083436075888182 trainer/global_step‚1749 % _timestamp‚1.7843418441377277e+09 _runtime‚1502.831006805 1loss/FBt1/assignment_loss‚0.08817449960301271†§ž· ‚(:“ àÊëÒ²»¢” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.697' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.339400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¥z©Å¸ ‚(:º àÊëÒ–# disk.nvme0c0n1.in‚ 3526.19921875& disk.nvme0c0n1.out‚5626.9169921875 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚439485 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚26.31084781462044 disk.sda3.out‚ 37.23046875$ memory_percent‚95.46902179718018* disk./.usagePercent‚40.097889191255405 disk.dm-0.in‚ 3526.19921875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 2.642318e+06 disk.rbd1.out‚ 3.89453125 disk.dm-1.in‚0* proc.memory.percent‚16.603612899780273$ disk./.usageGB‚597.4875259399414! disk.dm-0.out‚5626.9169921875 disk.rbd0.in‚0( proc.memory.availableMB‚ 742.35546875 proc.cpu.threads‚17$ disk.nvme0n1.out‚5626.9169921875 disk.rbd0.out‚0 disk.dm-1.out‚ 37.23046875" proc.memory.rssMB‚ 2720.3359375 disk.sda.in‚0 disk.sda.out‚ 37.23046875! disk.nvme0n1.in‚ 3526.19921875-A;½¹ ‚(:’ ïÊëÒø’¡” gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áº ‚(:¶ ïÊëÒ‘ø³– network.recv‚443869 disk.dm-1.out‚ 37.51171875 disk.sda.out‚ 37.51171875 disk.rbd1.out‚ 3.89453125* proc.memory.percent‚16.603612899780273 network.sent‚ 2.705232e+06 disk.drbd2165.in‚0! disk.dm-0.out‚5638.4033203125 disk.nvme0n1.in‚ 3527.1640625( proc.memory.availableMB‚ 742.44140625" proc.memory.rssMB‚ 2720.3359375 cpu‚26.305130769433678( disk./.usagePercent‚40.0978405498239 disk.rbd0.in‚0 disk.sda2.in‚0$ disk./.usageGB‚597.4868011474609 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 3527.1640625$ disk.nvme0n1.out‚5638.4033203125 disk.sda3.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚5638.4033203125 disk.dm-0.in‚ 3527.1640625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 37.51171875 disk.sda2.out‚0$ memory_percent‚95.46849727630615 proc.cpu.threads‚17Àù¹ì2» - _wandbruntime‚1542  _runtime‚1542Ò•M–Ǽ Á _runtime‚1542.286292582 _step‚144 & lr-AdamW/pg1‚0.0004434553001524741 & lr-AdamW/pg2‚0.0004434553001524741 trainer/global_step‚1799 % _timestamp‚1.7843418835925567e+09¸O B̽ Æ _runtime‚1542.286292582 _step‚144 & lr-AdamW/pg1‚0.0004434553001524741 & lr-AdamW/pg2‚0.0004434553001524741 trainer/global_step‚1799 % _timestamp‚1.7843418835925567e+09\`32¾ - _wandbruntime‚1542  _runtime‚15429óÀ:Ú'¿ ÔO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3864982 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1896903208960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3156961 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62278174523392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1896561869824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 _step‚145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚1542.287607976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272787943882752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3156958 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30862479 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273282792073214 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274684864131072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273283149662464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843418835938716e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1896903208960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3156961 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62278174523392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1896561869824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 _step‚145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚1542.287607976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272787943882752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3156958 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30862479 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273282792073214 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274684864131072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273283149662464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843418835938716e+09 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitoRn¹ r.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3158706 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62312737413632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274832759409152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919835847168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1542.903594812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3158706 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62312737413632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274832759409152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919835847168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚1542.903594812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30879163 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144’ÔLé*2Ä - _wandbruntime‚1542  _runtime‚1542ÍCixÅ € /loss/FBt1/detection_loss‚0.8996504994260298 /loss/FBt2/detection_loss‚0.4754452683055439 _runtime‚ 1542.90552168 0loss/FRt1/assignment_loss‚2.0256382187683424 2loss/SRqqt2/assignment_loss‚0.5887539365744284 1loss/FBt1/assignment_loss‚0.13431314776170364 epoch‚4 trainer/global_step‚1799 0loss/FRt2/assignment_loss‚0.9952463796325758 1loss/FBt2/assignment_loss‚0.09794654955703622 1loss/SRqqt1/detection_loss‚1.1166548356413841 _step‚147 /loss/FRt1/detection_loss‚0.5905067796737787 1loss/SRqqt2/detection_loss‚0.7282312237001088 %loss/total_loss‚9.632492005951892 % _timestamp‚1.7843418842122073e+09 2loss/SRqqt1/assignment_loss‚0.7515770865667408 /loss/FRt2/detection_loss‚1.2285280803442185;« Æ … 2loss/SRqqt1/assignment_loss‚0.7515770865667408 /loss/FRt2/detection_loss‚1.2285280803442185 /loss/FBt1/detection_loss‚0.8996504994260298 /loss/FBt2/detection_loss‚0.4754452683055439 _runtime‚ 1542.90552168 0loss/FRt1/assignment_loss‚2.0256382187683424 2loss/SRqqt2/assignment_loss‚0.5887539365744284 1loss/FBt1/assignment_loss‚0.13431314776170364 epoch‚4 trainer/global_step‚1799 0loss/FRt2/assignment_loss‚0.9952463796325758 1loss/FBt2/assignment_loss‚0.09794654955703622 1loss/SRqqt1/detection_loss‚1.1166548356413841 _step‚147 /loss/FRt1/detection_loss‚0.5905067796737787 1loss/SRqqt2/detection_loss‚0.7282312237001088 %loss/total_loss‚9.632492005951892 % _timestamp‚1.7843418842122073e+09“õüM¬“Ç ‚(:ˆ þÊëÒÍÙÔ“ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚152.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—„>¶ÄÈ ‚(:¹ þÊëÒ…±– cpu‚26.299523003869336 disk.drbd2165.in‚0$ memory_percent‚95.46735286712646$ disk./.usageGB‚597.4871215820312 disk.sda.out‚ 37.69140625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 37.69140625* disk./.usagePercent‚40.097862054456776 disk.sda.in‚0 disk.rbd1.out‚ 3.95703125 disk.rbd0.in‚0 network.recv‚447232# disk.nvme0c0n1.in‚ 3610.05859375 disk.rbd0.out‚0) proc.memory.percent‚16.60017967224121 network.sent‚ 2.71011e+06! disk.dm-0.out‚5643.4482421875 disk.sda2.in‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚5643.4482421875$ disk.nvme0n1.out‚5643.4482421875 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2719.7734375 disk.sda3.out‚ 37.69140625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 742.62890625 disk.dm-0.in‚ 3610.05859375 disk.sda2.out‚0! disk.nvme0n1.in‚ 3610.05859375—{߃žÉ ‚(:“ ËëÒäÚß” gpu.0.gpu‚54.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Håm#·Ê ‚(:¬ ËëÒáÛä• disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.2943549558678 proc.cpu.threads‚17 network.recv‚451868 disk.drbd2165.in‚0 network.sent‚ 2.773123e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 3610.890625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 3610.890625" proc.memory.rssMB‚ 2720.3359375) disk./.usagePercent‚40.09785386221568$ disk./.usageGB‚597.4869995117188 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.95703125 disk.dm-0.in‚ 3610.890625& proc.memory.availableMB‚ 742.578125* proc.memory.percent‚16.603612899780273$ disk.nvme0n1.out‚5653.7822265625 disk.sda3.out‚ 38.0078125 disk.drbd2165.out‚0 disk.dm-1.out‚ 38.0078125& disk.nvme0c0n1.out‚5653.7822265625# memory_percent‚95.4676628112793 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚5653.7822265625 disk.sda.out‚ 38.0078125 disk.sda3.in‚0Éjà©“Ë ‚(:ˆ œËëÒ‘²à“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éYTÔ¿Ì ‚(:´ œËëÒ×±– disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 3611.42578125 disk.sda2.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.01953125$ proc.memory.availableMB‚742.5625* proc.memory.percent‚16.603612899780273 cpu‚26.289092822980844 network.recv‚455248 proc.cpu.threads‚17" proc.memory.rssMB‚ 2720.3359375) disk./.usagePercent‚40.09783286959787 disk.drbd2165.out‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 3611.42578125$ disk.nvme0n1.out‚5659.2060546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 38.47265625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 3611.42578125 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚95.46775817871094& disk.nvme0c0n1.out‚5659.2060546875! disk.dm-0.out‚5659.2060546875 disk.sda.out‚ 38.47265625 disk.sda3.out‚ 38.47265625 network.sent‚ 2.77809e+06# disk./.usageGB‚597.486686706543LèŸ2Í - _wandbruntime‚1582  _runtime‚1582-ŒÎÉÎ Ã _step‚148 ' lr-AdamW/pg1‚0.00044304812641012643 ' lr-AdamW/pg2‚0.00044304812641012643 trainer/global_step‚1849 % _timestamp‚1.7843419237014039e+09 _runtime‚1582.394927502(K5 ÎÏ È trainer/global_step‚1849 % _timestamp‚1.7843419237014039e+09 _runtime‚1582.394927502 _step‚148 ' lr-AdamW/pg1‚0.00044304812641012643 ' lr-AdamW/pg2‚0.00044304812641012643”çdy2Ð - _wandbruntime‚1582  _runtime‚1582íV;ÏÑ ÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1944620107004 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62030811940864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7843419237024407e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947598710784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1944279557114 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282097611114496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3244211 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14605724 trainer/global_step‚1849 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IÅè StatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3244208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000760565248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947598710784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17102784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚ 1582.39612792 _step‚149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1A9MÞ'Ò ØO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚ 1582.39612792 _step‚149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1944620107004 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62030811940864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7843419237024407e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947598710784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1944279557114 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282097611114496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3244211 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14605724 trainer/global_step‚1849 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1948611344896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280297236393984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1945631160838 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17111850 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 % _timestamp‚1.7843419243169613e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1583.010725149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280297253433344 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0o¢+é&Õ ãM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1948611344896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280297236393984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1945631160838 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17111850 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 % _timestamp‚1.7843419243169613e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1583.010725149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280297253433344 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169–ßIþ2Ö - _wandbruntime‚1583  _runtime‚1583hT?ȶ× ü 0loss/FBt1/assignment_loss‚0.1251091975952737 0loss/FBt2/assignment_loss‚0.0755622906987884 1loss/SRqqt1/detection_loss‚1.0586155837793032 $ _timestamp‚1.784341924319968e+09 _runtime‚1583.013279384 0loss/FRt2/assignment_loss‚1.1160184304714202 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7353059466105252 /loss/FRt2/detection_loss‚1.2671121128059457 trainer/global_step‚1849 2loss/SRqqt2/assi.yqÿLgnment_loss‚0.5842832960636215 /loss/FRt1/detection_loss‚0.6764312797844375 0loss/SRqqt2/detection_loss‚0.638253040905813 .loss/FBt1/detection_loss‚0.958727492117568 /loss/FBt2/detection_loss‚0.5497070213082857 %loss/total_loss‚9.759636547545485 _step‚151 0loss/FRt1/assignment_loss‚1.9745108554045023lm)Ø  .loss/FBt1/detection_loss‚0.958727492117568 /loss/FBt2/detection_loss‚0.5497070213082857 %loss/total_loss‚9.759636547545485 _step‚151 0loss/FRt1/assignment_loss‚1.9745108554045023 0loss/FBt1/assignment_loss‚0.1251091975952737 0loss/FBt2/assignment_loss‚0.0755622906987884 1loss/SRqqt1/detection_loss‚1.0586155837793032 $ _timestamp‚1.784341924319968e+09 _runtime‚1583.013279384 0loss/FRt2/assignment_loss‚1.1160184304714202 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7353059466105252 /loss/FRt2/detection_loss‚1.2671121128059457 trainer/global_step‚1849 2loss/SRqqt2/assignment_loss‚0.5842832960636215 /loss/FRt1/detection_loss‚0.6764312797844375 0loss/SRqqt2/detection_loss‚0.638253040905813—\‚Çí“Ù ‚(:ˆ «ËëÒƒ­þ• gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.768' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.3536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÿ-ÀÚ ‚(:µ «ËëÒÙŽš— disk.rbd0.in‚0! disk.nvme0n1.in‚ 3611.85546875# disk.nvme0c0n1.in‚ 3611.85546875$ memory_percent‚95.46995162963867& proc.memory.availableMB‚ 742.203125* proc.memory.percent‚16.603612899780273 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 2.79606e+06* disk./.usagePercent‚40.097857702328696$ disk./.usageGB‚597.4870567321777 disk.rbd1.in‚0 disk.dm-0.in‚ 3611.85546875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 38.7890625 cpu‚26.283755146835013 disk.sda3.in‚0 disk.dm-1.out‚ 38.7890625! disk.dm-0.out‚5667.0615234375 disk.sda.out‚ 38.7890625" proc.memory.rssMB‚ 2720.3359375 disk.rbd1.out‚ 4.01953125 disk.sda1.out‚0 network.recv‚458969$ disk.nvme0n1.out‚5667.0615234375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚5667.0615234375 proc.cpu.threads‚17œJ¬ “Û ‚(:ˆ ºËëÒû©á“ gpu.0.gpu‚46.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BY7¾Ü ‚(:³ ºËëÒ¼Û²– disk.rbd1.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 3633.95703125! disk.dm-0.out‚5674.9189453125 disk.sda2.in‚0 disk.sda3.out‚ 38.984375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3633.95703125 disk.sda.in‚0( proc.memory.availableMB‚ 742.98828125) proc.memory.percent‚16.60017967224121 network.sent‚ 2.845857e+06 disk.rbd1.out‚ 4.08203125 disk.sda1.in‚0 disk.dm-1.out‚ 38.984375 disk.sda.out‚ 38.984375& disk.nvme0c0n1.out‚5674.9189453125! disk.nvme0n1.in‚ 3633.95703125 disk.sda3.in‚0$ memory_percent‚95.46515941619873 cpu‚26.27857606657562" proc.memory.rssMB‚ 2719.7734375 network.recv‚463482$ disk.nvme0n1.out‚5674.9189453125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0* disk./.usagePercent‚40.097851302140334$ disk./.usageGB‚597.4869613647461 disk.sda2.out‚0E³(¾Ý ‚(:³ ÉËëÒ’¾š• disk.rbd1.out‚ 4.08203125 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚95.46990394592285 disk.rbd1.in‚0 disk.sda1.out‚0 network.sent‚ 2.850893e+06$ disk./.usageGB‚597.4869728088379% disk.nvme0n1.out‚5681.01806640625 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚40.09785207016294 disk.drbd2165.out‚0 disk.dm-0.in‚ 3736.94140625" disk.dm-0.out‚5681.01806640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚466701 disk.sda2.in‚0* proc.memory.percent‚16.603612899780273 cpu‚26.273503393750854 disk.dm-1.in‚0 disk.dm-1.out‚39.34375# disk.nvme0c0n1.in‚ 3736.94140625' proc.memory.availableMB‚ 742.2109375" proc.memory.rssMB‚ 2720.3359375! disk.nvme0n1.in‚ 3736.94140625 disk.sda3.out‚39.34375' disk.nvme0c0n1.out‚5681.01806640625 disk.sda.out‚39.34375QXïžÞ ‚(:“ ÉËëÒÏé’• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š;qÂ2ß - _wandbruntime‚1622  _runtime‚1622íÚ\6Çà Á & lr-AdamW/pg1‚0.0004426269182009515 & lr-AdamW/pg2‚0.0004426269182009515 trainer/global_step‚1899 % _timestamp‚1.7843419635546584e+09 _runtime‚1622.248253216 _step‚152R»<áÌá Æ _runtime‚1622.248253216 _step‚152 & lr-AdamW/pg1‚0.0004426269182009515 & lr-AdamW/pg2‚0.0004426269182009515 trainer/global_step‚1899 % _timestamp‚1.7843419635546584e+09˜Þ­ŒÎ2â - _wandbruntime‚1622  _runtime‚1622’/øÚ'ã ÔO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286040213229568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17547259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1997952873472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287512063885312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚153 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1998294212608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952a$üQß'ä ÙO % _timestamp‚1.7843419635557604e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287512063885312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚153 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1998294212608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286040213229568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17547259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1997952873472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17556311 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1899 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14994956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63735678625792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63735697238016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17556311 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 [GDeviceStatsMonitor.on_train_batch_end/active_bytÏ'CA‰ es.large_pool.allocated‚287659304914944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17556311 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1899 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14994956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63735678625792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63735697238016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17556311 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287659304914944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67445861822464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18011688 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67445973206016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15368926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294874115366912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4175874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2048989714432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2048648375296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2ªÙ¿pô ÙO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65373706405888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293365729783808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18001711 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2045512530930 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1662.298337071 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4176151 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295411242314738 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚1949 % _timestamp‚1.7843420036042373e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚157 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294874115366912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15368924 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293365746823168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15368924 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7594862 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67445861822464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_trainl†Ho_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18011688 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67445973206016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15368926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294874115366912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4175874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2048989714432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2048648375296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294874098327552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295411599903988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1662.912368815 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295021356396544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4178240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚158 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3420453 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843420042185705e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297071358745088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65407135164416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15376558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33387319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3420456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101%ϧQé&÷ ãM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3420453 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843420042185705e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297071358745088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65407135164416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15376558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33387319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3420456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚65407116552192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1662.912368815 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295021356396544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4178240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚158že…€D2ø - _wandbruntime‚1662  _runtime‚1662ÞB”ù ý _runtime‚ 1662.91479838 _step‚159 2loss/FBt1/assignment_loss‚0.037823852978723656 /loss/FRt1/detection_loss‚0.6701429347028336 % _timestamp‚1.7843420042215161e+09 0loss/FRt2/assignment_loss‚1.2463914050051004 2loss/SRqqt2/assignment_loss‚0.6036427902223884 1loss/SRqqt2/detection_loss‚0.7505045966261913 epoch‚4 trainer/global_step‚1949 /loss/FRt1/assignment_loss‚2.081680416316569 0loss/FBt2/assignment_loss‚0.0689813067807348 /loss/FBt1/detection_loss‚1.4764064215123653 0loss/SRqqt1/assignment_loss‚0.74119267328471 /loss/FRt2/detection_loss‚1.3631129086494815 0loss/SRqqt1/detection_loss‚1.069156044415108 /loss/FBt2/detection_loss‚0.4899514863539969 &loss/total_loss‚10.598986836848203¤ú ‚ 1loss/SRqqt2/detection_loss‚0.7505045966261913 epoch‚4 trainer/global_step‚1949 /loss/FRt1/assignment_loss‚2.081680416316569 0loss/FBt2/assignment_loss‚0.0689813067807348 /loss/FBt1/detection_loss‚1.4764064215123653 0loss/SRqqt1/assignment_loss‚0.74119267328471 /loss/FRt2/detection_loss‚1.3631129086494815 0loss/SRqqt1/detection_loss‚1.069156044415108 /loss/FBt2/detection_loss‚0.4899514863539969 &loss/total_loss‚10.598986836848203 _runtime‚ 1662.91479838 _step‚159 2loss/FBt1/assignment_loss‚0.037823852978723656 /loss/FRt1/detection_loss‚0.6701429347028336 % _timestamp‚1.7843420042215161e+09 0loss/FRt2/assignment_loss‚1.2463914050051004 2loss/SRqqt2/assignment_loss‚0.6036427902223884ŸŸ GÖ“û ‚(:ˆ öËëÒèñÅ“ gpu.0.gpu‚48.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¾}AÃü ‚(:¸ öËëÒ¿êè• disk.sda.in‚0 disk.sda2.out‚0$ disk./.usageGB‚597.4838562011719( proc.memory.availableMB‚ 496.19140625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 40.734375# disk.nvme0c0n1.in‚ 3738.73828125' disk.nvme0c0n1.out‚5703.12451171875 disk.dm-0.in‚ 3738.73828125# memory_percent‚96.9714879989624! disk.nvme0n1.in‚ 3738.73828125# proc.memory.rssMB‚ 2720.33984375* disk./.usagePercent‚40.097642912007466" disk.dm-0.out‚5703.12451171875 disk.sda.out‚ 40.734375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.26953125 network.recv‚477992 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚5703.12451171875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 40.734375* proc.memory.percent‚16.603636741638184 cpu‚26.258689126605127 network.sent‚ 2.923622e+06ö`¢žý ‚(:“ …ÌëÒËîð“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¬„Àþ ‚(:µ …ÌëÒ”ßÖ disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 3842.34765625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2720.33984375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚5718.16357421875 network.sent‚ 2.986623e+06$ disk./.usageGB‚597.4839935302734 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 3842.34765625 disk.sda.out‚42.40625% disk.nvme0n1.out‚5718.16357421875 disk.sda3.in‚0$ memory_percent‚96.97470664978027 disk.sda2.out‚0 cpu‚26.253908327190516" disk.dm-0.out‚5718.16357421875 disk.dm-1.out‚42.40625 disk.rbd1.out‚ 4.26953125* proc.memory.percent‚16.603636741638184 disk.dm-0.in‚ 3842.34765625' proc.memory.availableMB‚ 495.6640625 disk.sda3.out‚42.40625 network.recv‚482687* disk./.usagePercent‚40.097652128278696 disk.rbd0.out‚0 disk.drbd2165.in‚0‰ü’ÿ ‚(:’ ”ÌëÒ¤µß“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žRu¼€ ‚(:± ”ÌëÒõ©œ– network.recv‚485392 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3843.1796875$ memory_percent‚96.97167873382568% disk.nvme0c0n1.out‚5726.029296875 network.sent‚ 2.991424e+06 disk.sda1.in‚0 disk.sda3.out‚ 43.1796875 disk.dm-1.out‚ 43.1796875 disk.dm-0.out‚5726.029296875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 3843.1796875# disk.nvme0n1.out‚5726.029296875( proc.memory.availableMB‚ 496.16015625 cpu‚26.24895261011339 disk.rbd0.in‚0# proc.memory.rssMB‚ 2720.33984375$ disk./.usageGB‚597.4839782714844 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 3843.1796875 disk.sda2.in‚0 disk.rbd1.out‚ 4.33203125) disk./.usagePercent‚40.09765110424856 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 43.1796875 disk.rbd1.in‚0* proc.memory.percent‚16.603636741638184JG½ç2 - _wandbruntime‚1702  _runtime‚1702f3+É‚ à % _timestamp‚1.7843420436020503e+09 _runtime‚1702.295670798 _step‚160 ' lr-AdamW/pg1‚0.00044174250749545453 ' lr-AdamW/pg2‚0.00044174250749545453 trainer/global_step‚1999^%΃ È ' lr-AdamW/pg1‚0.00044174250749545453 ' lr-AdamW/pg2‚0.00044174250749545453 trainer/global_step‚1999 % _timestamp‚1.7843420436020503e+09 _runtime‚1702.295670798 _step‚160 ÖÞ82„ - _wandbruntime‚1702  _runtime‚1702õ€íÏ… ÔO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/®R—ž &inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302236149809152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2096469567728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18456163 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7786315 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302787392395246 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _runtime‚1702.296945253 _step‚161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4280354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304335852064768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18466140 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302236149809152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1999 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2099685216256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304335852064768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302787749984496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269G.žòß'† ÙO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302236149809152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2096469567728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18456163 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7786315 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302787392395246 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _runtime‚1702.296945253 _step‚161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4280354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304335852064768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18466140 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302236149809152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1999 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2099685216256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304335852064768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302787749984496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438¡¢©r2‡ - _wandbruntime‚1702  _runtime‚1702Ú|Ž£ä&ˆ ÞM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302383407878144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15758158 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15758156 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2100697850368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚1999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843420442170625e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 _runtime‚1702.910886267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302383407878144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302383390838784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3507706 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18475206 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124437030400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0V XÝ ‰ ãM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚1999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843420442170625e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 _runtime‚1702.910886267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302383407878144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302383390838784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3507706 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18475206 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124437030400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302383407878144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15758158 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15758156 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2100697850368¢ ]à 2Š - _wandbruntime‚1702  _runtime‚1702 “x‹ þ trainer/global_step‚1999 % _timestamp‚1.7843420442201786e+09 epoch‚4 /loss/FRt2/assignment_loss‚1.162406057006908 1loss/FBt1/assignment_loss‚0.06946308187481265 .loss/FBt2/detection_loss‚0.476183319145621 _runtime‚ 1702.91350994 _step‚163 0loss/FRt1/assignment_loss‚1.9535601809295504 2loss/SRqqt2/assignment_loss‚0.6042205671942167 1loss/FBt2/assignment_loss‚0.03254326327994428 1loss/SRqqt2/detection_loss‚0.7039850934071379 %loss/total_loss‚9.221319239568267 2loss/SRqqt1/assignment_loss‚0.6301394429947562 /loss/FRt1/detection_loss‚0.6106073913472648 /loss/FRt2/detection_loss‚1.2764387737524714 1loss/SRqqt1/detection_loss‚1.0150431559672335 /loss/FBt1/detection_loss‚0.68672891266835052ø€Ó Œ ƒ 0loss/FRt1/assignment_loss‚1.9535601809295504 2loss/SRqqt2/assignment_loss‚0.6042205671942167 1loss/FBt2/assignment_loss‚0.03254326327994428 1loss/SRqqt2/detection_loss‚0.7039850934071379 %loss/total_loss‚9.221319239568267 2loss/SRqqt1/assignment_loss‚0.6301394429947562 /loss/FRt1/detection_loss‚0.6106073913472648 /loss/FRt2/detection_loss‚1.2764387737524714 1loss/SRqqt1/detection_loss‚1.0150431559672335 /loss/FBt1/detection_loss‚0.6867289126683505 trainer/global_step‚1999 % _timestamp‚1.7843420442201786e+09 epoch‚4 /loss/FRt2/assignment_loss‚1.162406057006908 1loss/FBt1/assignment_loss‚0.06946308187481265 .loss/FBt2/detection_loss‚0.476183319145621 _runtime‚ 1702.91350994 _step‚163£,öŠ¸œ ‚(:‘ £Ìë҆ĭ• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¹¤Ž ‚(:™ £Ìëүܺ– disk.rbd1.in‚0$ memory_percent‚96.97372913360596* disk./.usagePercent‚40.097675424964315 disk.sda1.in‚0( proc.memory.availableMB‚ 495.82421875 network.recv‚488562 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 3843.1796875 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.out‚ 5733.984375$ disk./.usageGB‚597.4843406677246 disk.dm-0.out‚ 5733.984375 disk.sda3.in‚0 disk.sda3.out‚43.75 disk.rbd1.out‚ 4.33203125 disk.sda2.in‚0" disk.nvme0c0n1.out‚ 5733.984375 network.sent‚ 2.99562e+06 disk.dm-1.out‚43.75 disk.drbd2165.in‚0 disk.sda.out‚43.75# proc.memory.rssMB‚ 2720.33984375 cpu‚26.25945060777941 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 3843.1796875* proc.memory.percent‚16.603636741638184 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 3843.1796875ã°Z!2 - _wandbruntime‚1720  _runtime‚1720Û )à= Ú{ 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17398883402347565 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8397549390792847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7631428837776184 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8257868885993958 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9772137999534607 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19148647785186768 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6763350367546082 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6507223844528198 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7270996570587158 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48236310482025146 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07456382364034653 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7257571220397949 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20888611674308777 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8375845551490784 'particle/f_score‚0.6155542135238647 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28637927770614624 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6090690493583679 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07271905988454819 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6099706888198853 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9937736988067627 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2150394171476364 /particle/accuracy_4_of_4‚0.9508711695671082 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25925925374031067 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923270344734192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6335035562515259 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193820476531982 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23447391390800476 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6671520471572876 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9812041521072388 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3763301372528076 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20924651622772217 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5162500143051147 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6363967061042786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10003737360239029 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9640812873840332 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912372827529907 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8978353142738342 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚164 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9890828132629395 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6791059970855713 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3733396530151367 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6321433186531067 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890228509902954 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529557824134827 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6335035562515259 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18568097054958344 $ _timestamp‚1.784342061571005e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6773270964622498 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9868394732475281 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23338568210601807 )jet/accuracy_2_of_4‚0.733440101146698 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6809033751487732 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4017462134361267 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31748002767562866 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020122677087783813 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06832505762577057 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24717815220355988 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6089377403259277 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7053357362747192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026718977838754654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9688616991043091 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9952741265296936 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913339018821716 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8103930950164795 *jet/accuracy_1_of_2‚0.7737891674041748 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7339840531349182 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7validation_average_jet_accuracy‚0.36542248725891113 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21678869426250458 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1306389719247818 trainer/global_step‚1999 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3906606435775757 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6713282465934753 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 )loss/val_total_loss‚9.484977666387355 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05097823590040207 /particle/accuracy_1_of_2‚0.9998962879180908 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6210731863975525 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6197940111160278 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7096467018127441 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.05747527256608009 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.32828766107559204 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7291395664215088 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9937736988067627 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06832505762577057 *jet/accuracy_1_of_1‚0.3284105360507965 /particle/accuracy_2_of_2‚0.9998962879180908 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9942193627357483 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22746065258979797 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9814193248748779 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17718875408172607 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23599517345428467 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7356065511703491 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.791269838809967 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24174213409423828 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957523226737976 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.769066572189331 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18037636578083038 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23599517345428467 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03834544122219086 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.986273467540741 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25925925374031067 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6330715417861938 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7297680974006653 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13358265161514282 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6860207319259644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8079554438591003 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0414179265499115 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24609427154064178 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9882766008377075 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19728828966617584 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ,jet/accuracy_3_of_4‚0.055419325828552246 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5457060933113098 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2905607521533966 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9764779210090637 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.631512463092804 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11755639314651489 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8016453981399536 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8016453981399536 +particle/specificity‚0.9010583758354187 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8358648419380188 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04825447127223015 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07353992760181427 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6054887771606445 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30136480927467346 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2062583714723587 _runtime‚1720.265333148 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0721563771367073 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5558941960334778 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2308499962091446 *jet/accuracy_2_of_3‚0.5002991557121277 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19660672545433044 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.973382830619812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7050642967224121 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.37648606300354004 *jet/accuracy_1_of_3‚0.9098111987113953 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6373398900032043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20655237138271332 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6447703242301941 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8976157903671265 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7624433636665344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7996599078178406 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11592400819063187 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 epoch‚4 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6335889101028442 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42424482107162476 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6771791577339172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4810849130153656/7ýl$‘ ß{ 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23338568210601807 )jet/accuracy_2_of_4‚0.733440101146698 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6809033751487732 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4017462134361267 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31748002767562866 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020122677087783813 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06832505762577057 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24717815220355988 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6089377403259277 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7053357362747192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026718977838754654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9688616991043091 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9952741265296936 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913339018821716 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8103930950164795 *jet/accuracy_1_of_2‚0.7737891674041748 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7339840531349182 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7validation_average_jet_accuracy‚0.36542248725891113 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21678869426250458 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1306389719247818 trainer/global_step‚1999 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3906606435775757 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6713282465934753 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 )loss/val_total_loss‚9.484977666387355 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05097823590040207 /particle/accuracy_1_of_2‚0.9998962879180908 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6210731863975525 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6197940111160278 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7096467018127441 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.05747527256608009 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.32828766107559204 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7291395664215088 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9937736988067627 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06832505762577057 *jet/accuracy_1_of_1‚0.3284105360507965 /particle/accuracy_2_of_2‚0.9998962879180908 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9942193627357483 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22746065258979797 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9814193248748779 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17718875408172607 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23599517345428467 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7356065511703491 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.791269838809967 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24174213409423828 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957523226737976 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.769066572189331 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18037636578083038 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23599517345428467 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03834544122219086 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.986273467540741 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.307692vóMñÁ#3191547394 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25925925374031067 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6330715417861938 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7297680974006653 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13358265161514282 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6860207319259644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8079554438591003 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0414179265499115 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24609427154064178 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9882766008377075 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19728828966617584 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ,jet/accuracy_3_of_4‚0.055419325828552246 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5457060933113098 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2905607521533966 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9764779210090637 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.631512463092804 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11755639314651489 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8016453981399536 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8016453981399536 +particle/specificity‚0.9010583758354187 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8358648419380188 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04825447127223015 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07353992760181427 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6054887771606445 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30136480927467346 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2062583714723587 _runtime‚1720.265333148 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0721563771367073 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5558941960334778 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2308499962091446 *jet/accuracy_2_of_3‚0.5002991557121277 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19660672545433044 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.973382830619812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7050642967224121 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.37648606300354004 *jet/accuracy_1_of_3‚0.9098111987113953 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6373398900032043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20655237138271332 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6447703242301941 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8976157903671265 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7624433636665344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7996599078178406 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11592400819063187 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 epoch‚4 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6335889101028442 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42424482107162476 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6771791577339172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4810849130153656 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17398883402347565 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8397549390792847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7631428837776184 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8257868885993958 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9772137999534607 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19148647785186768 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6763350367546082 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6507223844528198 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7270996570587158 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48236310482025146 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07456382364034653 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7257571220397949 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20888611674308777 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8375845551490784 'particle/f_score‚0.6155542135238647 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28637927770614624 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6090690493583679 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07271905988454819 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6099706888198853 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9937736988067627 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2150394171476364 /particle/accuracy_4_of_4‚0.9508711695671082 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25925925374031067 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923270344734192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6335035562515259 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193820476531982 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23447391390800476 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6671520471572876 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9812041521072388 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3763301372528076 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20924651622772217 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5162500143051147 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6363967061042786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10003737360239029 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9640812873840332 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912372827529907 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8978353142738342 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚164 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9890828132629395 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6791059970855713 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3733396530151367 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6321433186531067 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890228509902954 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529557824134827 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6335035562515259 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18568097054958344 $ _timestamp‚1.784342061571005e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6773270964622498 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9868394732475281 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442¤Wƒaˆ“’ ‚(:ˆ ²ÌëÒþ¥Á“ gpu.0.gpu‚93.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09õÇ=¸“ ‚(:­ ²ÌëҚɅ– disk.sda1.out‚0' proc.memory.availableMB‚ 742.1640625) disk./.usagePercent‚40.09766390462527 disk.rbd1.in‚0 disk.drbd2165.in‚0 network.sent‚ 3.05857e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 44.49609375 disk.dm-0.in‚ 3844.14453125 disk.sda.in‚0 disk.sda3.out‚ 44.49609375 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3844.14453125# proc.memory.rssMB‚ 2714.73046875$ memory_percent‚95.47019004821777$ disk./.usageGB‚597.4841690063477 disk.dm-0.out‚ 5988.8984375 disk.sda2.in‚0 disk.rbd1.out‚ 4.39453125# disk.nvme0c0n1.out‚ 5988.8984375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0( proc.memory.percent‚16.5693998336792 cpu‚26.27648872640583 disk.sda.out‚ 44.49609375 disk.sda3.in‚0 network.recv‚493230! disk.nvme0n1.in‚ 3844.14453125! disk.nvme0n1.out‚ 5988.8984375 disk.rbd0.in‚0'ž” ‚(:“ ÁÌëÒìÆÛ“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O*B¾• ‚(:³ ÁÌëÒÞ¢û•! disk.nvme0c0n1.in‚ 3943.953125' disk.nvme0c0n1.out‚6058.93212890625 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.39453125# proc.memory.rssMB‚ 2714.73046875 disk.dm-1.out‚ 44.7265625 disk.drbd2165.in‚0 disk.dm-0.in‚ 3943.953125 disk.sda2.out‚0# memory_percent‚95.4702615737915 disk.drbd2165.out‚0 network.recv‚497597* disk./.usagePercent‚40.097577630086235 disk.sda.in‚0 disk.nvme0n1.in‚ 3943.953125% disk.nvme0n1.out‚6058.93212890625" disk.dm-0.out‚6058.93212890625 disk.sda3.out‚ 44.7265625 network.sent‚ 3.115939e+06 disk.sda1.in‚0 disk.sda.out‚ 44.7265625( proc.memory.percent‚16.5693998336792 cpu‚26.271461202759863 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚597.4828834533691 disk.dm-1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 742.15234375 disk.rbd0.out‚0ÞmÇ)“– ‚(:ˆ ÐÌëÒ¬ÛÞ“ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚246.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÍóÇ— ‚(:¼ ÐÌëÒÉÞ„˜) disk./.usagePercent‚40.09758940643281 disk.dm-1.out‚ 45.12890625$ memory_percent‚95.46878337860107( proc.memory.percent‚16.5693998336792 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3944.78515625 disk.dm-0.in‚ 3944.78515625 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚6247.89306640625# proc.memory.rssMB‚ 2714.73046875 cpu‚26.266872141492907 disk.dm-1.in‚0( proc.memory.availableMB‚ 742.39453125 disk.drbd2165.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚597.4830589294434 disk.sda.out‚ 45.12890625! disk.nvme0n1.in‚ 3944.78515625% disk.nvme0n1.out‚6247.89306640625 disk.sda3.out‚ 45.12890625 disk.sda2.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚6247.89306640625 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.45703125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.120793e+06 network.recv‚500948 gzÇ2˜ - _wandbruntime‚1761  _runtime‚1761žžçrÉ™ à ' lr-AdamW/pg2‚0.00044127936180317947 trainer/global_step‚2049 % _timestamp‚1.7843421030063252e+09 _runtime‚1761.699877223 _step‚165 ' lr-AdamW/pg1‚0.00044127936180317947dL•Îš È ' lr-AdamW/pg1‚0.00044127936180317947 ' lr-AdamW/pg2‚0.00044127936180317947 trainer/global_step‚2049 % _timestamp‚1.7843421030063252e+09 _runtime‚1761.699877223 _step‚165¥~ã2› - _wandbruntime‚1761  _runtime‚1761+’™Ú'œ ÔO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16252792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311527784980480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18980942 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3595728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18990919 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313695540528128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2164805779870 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18980942 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2164465229980 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35233732 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313695898906624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313695898906624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70925211819520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309925460185088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 _step‚166 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312089925415068 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68732475746304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2167772587008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167772587008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2192736073216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252790 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4414482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414742 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309925477224448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18990919 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311527767941120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311527784980480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312089925415068 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68732475746304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2167772587008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167772587008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2192736073216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252790 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4414482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414742 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309925477224448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18990919 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311527767941120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311527784980480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311527784980480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18980942 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3595728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18990919 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313695540528128¦’•èd2ž - _wandbruntime‚1762  _runtime‚1762Бaä&Ÿ ÞM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2193843304448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311675026010112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚1762.316034059 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68765923117056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 _step‚167 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0AÌOÈé&  ãM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 _step‚167 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2193843304448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311675026010112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚1762.316034059 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68765923117056 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDe}f ¯+viceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚170 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚2099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚170 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚2099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2099 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8204825 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219480715776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19454437 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219822062080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70437342431232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317397521489920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16642024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72682404278272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616º`ž¦¾² âM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219480715776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19454437 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219822062080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70437342431232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317397521489920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16642024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72682404278272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 XJDeviceStatsMonibuœ¬* tor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219822062080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3684723 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 $ _timestamp‚1.784342143692462e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2099 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8204825 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976«–" ‡2³ - _wandbruntime‚1802  _runtime‚1802¥–ÀD´  /loss/FRt1/assignment_loss‚1.988409710493815 2loss/SRqqt2/assignment_loss‚0.6481685233460348 epoch‚5 trainer/global_step‚2099 % _timestamp‚1.7843421436954372e+09 _runtime‚1802.388724097 0loss/FRt2/assignment_loss‚1.2088633328676224 2loss/SRqqt1/assignment_loss‚0.6862834279200245 1loss/FBt2/assignment_loss‚0.05184591788947632 1loss/FBt1/assignment_loss‚0.07488966927593156 /loss/FRt1/detection_loss‚0.6320568130302296 /loss/FRt2/detection_loss‚1.3491838787116237 1loss/SRqqt1/detection_loss‚1.1265330262298956 _step‚172 1loss/SRqqt2/detection_loss‚0.6245081897382136 /loss/FBt1/detection_loss‚0.8589707054197788 0loss/FBt2/detection_loss‚0.48520203182733684 %loss/total_loss‚9.734915226749983ã%pÐ µ † 1loss/SRqqt2/detection_loss‚0.6245081897382136 /loss/FBt1/detection_loss‚0.8589707054197788 0loss/FBt2/detection_loss‚0.48520203182733684 %loss/total_loss‚9.734915226749983 /loss/FRt1/assignment_loss‚1.988409710493815 2loss/SRqqt2/assignment_loss‚0.6481685233460348 epoch‚5 trainer/global_step‚2099 % _timestamp‚1.7843421436954372e+09 _runtime‚1802.388724097 0loss/FRt2/assignment_loss‚1.2088633328676224 2loss/SRqqt1/assignment_loss‚0.6862834279200245 1loss/FBt2/assignment_loss‚0.05184591788947632 1loss/FBt1/assignment_loss‚0.07488966927593156 /loss/FRt1/detection_loss‚0.6320568130302296 /loss/FRt2/detection_loss‚1.3491838787116237 1loss/SRqqt1/detection_loss‚1.1265330262298956 _step‚172¬Ÿ¿p“¶ ‚(:ˆ ŒÍëÒŒ§• gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ÍØ»· ‚(:° ŒÍëÒ³„†– disk.sda3.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 3.198576e+06 network.recv‚515414# disk./.usageGB‚594.368839263916" disk.dm-0.out‚6312.11181640625 disk.drbd2165.in‚0 disk.rbd1.in‚0 cpu‚26.24875503777098 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 46.8984375* disk./.usagePercent‚39.888591511674925 disk.sda.in‚0 disk.nvme0n1.in‚ 4041.9921875$ memory_percent‚95.47080993652344$ proc.memory.availableMB‚742.0625) proc.memory.percent‚16.56942367553711 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 4041.9921875' disk.nvme0c0n1.out‚6312.11181640625 disk.sda.out‚ 46.8984375 disk.sda3.out‚ 46.8984375 disk.rbd1.out‚ 4.64453125 disk.rbd0.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2714.734375 disk.dm-0.in‚ 4041.9921875 disk.sda2.in‚0% disk.nvme0n1.out‚6312.11181640625ø¢Ò!¸ ‚(:„ ›ÍëÒ‚¯à“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å°À¹ ‚(:µ ›ÍëÒÁÜô• disk.nvme0n1.in‚ 4041.9921875% disk.nvme0n1.out‚6318.17333984375 disk.sda3.in‚0 network.recv‚519718 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.64453125& proc.memory.availableMB‚ 742.265625! proc.memory.rssMB‚ 2714.734375$ memory_percent‚95.46957015991211" disk.nvme0c0n1.in‚ 4041.9921875# disk./.usageGB‚594.369312286377 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 4041.9921875" disk.dm-0.out‚6318.17333984375 disk.sda.in‚0 cpu‚26.244464925449083 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6318.17333984375 disk.sda2.out‚0 disk.sda3.out‚ 47.24609375) proc.memory.percent‚16.56942367553711 proc.cpu.threads‚17 network.sent‚ 3.261425e+06) disk./.usagePercent‚39.88862325660917 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 47.24609375 disk.drbd2165.in‚0 disk.sda.out‚ 47.24609375T½é2º - _wandbruntime‚1841  _runtime‚1841_¯.ÅÉ» à ' lr-AdamW/pg1‚0.00044031133847257726 ' lr-AdamW/pg2‚0.00044031133847257726 trainer/global_step‚2149 % _timestamp‚1.7843421829154236e+09 _runtime‚1841.608970658 _step‚173ªCGqμ È ' lr-AdamW/pg1‚0.00044031133847257726 ' lr-AdamW/pg2‚0.00044031133847257726 trainer/global_step‚2149 % _timestamp‚1.7843421829154236e+09 _runtime‚1841.608970658 _step‚173­¯|ëó2½ - _wandbruntime‚1841  _runtime‚1841Á»¿QØ'¾ ÒO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 # _timestamp‚1.78434218291649e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015990 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8391425 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2269504929792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521392873472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326842583165334 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19899823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326251870904320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266038753686 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3770228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2295228423168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324576527372288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326251887943680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326251870904320 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚174 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72075370211328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324576544411648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326251887943680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2269504929792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19889846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4621194 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169…CŸÊÝ'¿ ×O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19889846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4621194 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 # _timestamp‚1.78434218291649e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015990 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8391425 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2269504929792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521392873472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326842583165334 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19899823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326251870904320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266038753686 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3770228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2295228423168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324576527372288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326251887943680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326251870904320 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚174 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72075370211328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324576544411648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326251887943680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2269504929792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154®¬ÚÆ2À - _wandbruntime‚1842  _runtime‚1842ìèw¢Á ÞM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266709257518 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_baKEüBtch_end/active.small_pool.allocated‚19908889 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19908889 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚326399128973312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326989747301678 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2270517563904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚1842.224269062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328669646537216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326990104891040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872ÄŽôé& ãM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19908889 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19908889 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚326399128973312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326989747301678 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2270517563904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚1842.224269062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328669646537216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326990104891040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266709257518 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843422229380853e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4723773 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚1881.631833084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17397592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20344298 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319859092480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4723509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20344298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚178 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320200431616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76092904985088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843422229380853e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481²wK2Ò - _wandbruntime‚1882  _runtime‚1882ÅB?ã&Ó ÝM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8585045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784342223554579e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332048588677120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4725548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333761163415552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336082393520640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2317666294428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334366254971548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1882.248423303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333761180454912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 _step‚179]Ϙè&Ô âM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 _step‚179 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8585045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784342223554579e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332048588677120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4725548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333761163415552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336082393520640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2317666294428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334366254971548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1882.248423303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333761180454912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0³ŒŠp2Õ - _wandbruntime‚1882  _runtime‚1882ѾÈ Ö ƒ 1loss/SRqqt1/detection_loss‚0.9613818717012643 /loss/FBt1/detection_loss‚1.1281861488521099 % _timestamp‚1.7843422235566468e+09 _runtime‚1882.249918004 0loss/FRt1/assignment_loss‚1.8694268229550548 %loss/total_loss‚9.641982863509623 epoch‚5 _step‚180 2loss/SRqqt2/assignment_loss‚0.5733286118938566 .loss/FRt2/detection_loss‚1.342439776069662 1loss/SRqqt2/detection_loss‚0.7239276218311032 0loss/FBt2/detection_loss‚0.47498562943487277 trainer/global_step‚2199 2loss/FBt1/assignment_loss‚0.059436176910094216 2loss/FBt2/assignment_loss‚0.021616322975193347 /loss/FRt1/detection_loss‚0.6231614872721788 0loss/FRt2/assignment_loss‚1.0982859823230675 2loss/SRqqt1/assignment_loss‚0.7658064112911658e¾ÿ× ˆ 1loss/SRqqt2/detection_loss‚0.7239276218311032 0loss/FBt2/detection_loss‚0.47498562943487277 trainer/global_step‚2199 2loss/FBt1/assignment_loss‚0.059436176910094216 2loss/FBt2/assignment_loss‚0.021616322975193347 /loss/FRt1/detection_loss‚0.6231614872721788 0loss/FRt2/assignment_loss‚1.0982859823230675 2loss/SRqqt1/assignment_loss‚0.7658064112911658 1loss/SRqqt1/detection_loss‚0.9613818717012643 /loss/FBt1/detection_loss‚1.1281861488521099 % _timestamp‚1.7843422235566468e+09 _runtime‚1882.249918004 0loss/FRt1/assignment_loss‚1.8694268229550548 %loss/total_loss‚9.641982863509623 epoch‚5 _step‚180 2loss/SRqqt2/assignment_loss‚0.5733286118938566 .loss/FRt2/detection_loss‚1.342439776069662´ó²¸Ø ‚(:’ ×ÍëÒÏòù• gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¡C„»Ù ‚(:° ×ÍëÒ†©Ü—" disk./.usageGB‚594.36962890625$ disk.nvme0n1.out‚6351.2138671875 disk.rbd1.in‚0 network.sent‚ 3.339231e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 48.38671875! disk.nvme0c0n1.in‚ 4149.171875 disk.sda.in‚0( proc.memory.availableMB‚ 742.55078125 disk.rbd1.out‚ 4.76953125 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚6351.2138671875 disk.sda.out‚ 48.38671875 disk.nvme0n1.in‚ 4149.171875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4149.171875) proc.memory.percent‚16.56942367553711 cpu‚26.228039539709318) disk./.usagePercent‚39.88864450523452 disk.sda3.out‚ 48.38671875! proc.memory.rssMB‚ 2714.734375 disk.sda2.out‚0 disk.drbd2165.in‚0$ memory_percent‚95.46782970428467 network.recv‚534530 disk.sda1.in‚0! disk.dm-0.out‚6351.2138671875 proc.cpu.threads‚17,=Èü“Ú ‚(:ˆ æÍëÒôÔ” gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚(¡ÛîÛ ‚(:° æÍëÒ《– disk.sda3.in‚0 disk.sda3.out‚ 48.578125# proc.memory.rssMB‚ 2714.73828125 disk.rbd0.in‚0! disk.dm-0.out‚6358.8564453125 disk.sda2.in‚0 disk.rbd1.out‚ 4.83203125 network.sent‚ 3.4¤7·VÍ02332e+06( disk./.usagePercent‚39.8886406651215& disk.nvme0c0n1.out‚6358.8564453125 disk.sda.in‚0! disk.nvme0n1.in‚ 4150.13671875 disk.rbd1.in‚0 network.recv‚538984# memory_percent‚95.4704761505127' proc.memory.availableMB‚ 742.1171875 disk.rbd0.out‚0$ disk.nvme0n1.out‚6358.8564453125 cpu‚26.224118927598493 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 48.578125 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 4150.13671875# disk.nvme0c0n1.in‚ 4150.13671875 disk.sda.out‚ 48.578125) proc.memory.percent‚16.56944751739502# disk./.usageGB‚594.369571685791 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0-‰ÜnžÜ ‚(:“ õÍëÒÁ°ø” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.†“ÀÝ ‚(:µ õÍëÒ‹°–) disk./.usagePercent‚39.88864552926465' disk.nvme0c0n1.out‚6370.79150390625 disk.sda3.out‚ 49.7265625$ memory_percent‚95.47038078308105' proc.memory.availableMB‚ 742.1328125 proc.cpu.threads‚17$ disk./.usageGB‚594.3696441650391 disk.dm-1.in‚0) proc.memory.percent‚16.56944751739502 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4234.4765625 disk.dm-0.in‚ 4234.4765625 disk.sda.in‚0 network.sent‚ 3.407475e+06 network.recv‚542646 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 cpu‚26.21994825845044 disk.dm-1.out‚ 49.7265625" disk.dm-0.out‚6370.79150390625 disk.sda2.in‚0 disk.rbd1.out‚ 4.83203125 disk.rbd0.in‚0 disk.sda.out‚ 49.7265625 disk.nvme0n1.in‚ 4234.4765625% disk.nvme0n1.out‚6370.79150390625 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2714.73828125àbê2Þ - _wandbruntime‚1921  _runtime‚1921[2oÇß Á & lr-AdamW/pg2‚0.0004392878812798841 trainer/global_step‚2249 % _timestamp‚1.7843422628637257e+09 _runtime‚1921.557299112 _step‚181 & lr-AdamW/pg1‚0.0004392878812798841Õ#Ìà Æ _runtime‚1921.557299112 _step‚181 & lr-AdamW/pg1‚0.0004392878812798841 & lr-AdamW/pg2‚0.0004392878812798841 trainer/global_step‚2249 % _timestamp‚1.7843422628637257e+09µì¡#2á - _wandbruntime‚1921  _runtime‚1921i¾µÉÚ'â ÔO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20798750 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2366931177612 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75418246064128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340975990906880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272)Ù¬ß'ã ÙO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2366931177612 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75418246064128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340975990906880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20798750 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537¶/)ªÊ2ä - _wandbruntime‚1922  _runtime‚1922Ži—›¦%å ÝM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77850052153344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4828846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75451674822656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2371567221248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚2249 _runtime‚ 1922.17422533 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341123214897152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339374122270720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341123231936512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343494782118400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17786822 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17786824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3946476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚Ää4=20807802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438;òEåè&æ âM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3946476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20807802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77850052153344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4828846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75451674822656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2371567221248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚2249 _runtime‚ 1922.17422533 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341123214897152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339374122270720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341123231936512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343494782118400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17786822 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17786824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044·[^3€2ç - _wandbruntime‚1922  _runtime‚1922gä2è ÿ 0loss/FRt2/assignment_loss‚1.1258186544690814 .loss/FRt1/detection_loss‚0.616098086091726 1loss/SRqqt1/detection_loss‚1.1123864754776445 1loss/SRqqt2/detection_loss‚0.6836387345311264 2loss/SRqqt2/assignment_loss‚0.5956625080572552 # _timestamp‚1.78434226348264e+09 2loss/SRqqt1/assignment_loss‚0.6568273057105155 1loss/FBt2/assignment_loss‚0.07760440591814712 /loss/FRt2/detection_loss‚1.3616680835775214 /loss/FBt1/detection_loss‚1.2186744702530319 trainer/global_step‚2249 _runtime‚1922.175911654 _step‚184 0loss/FRt1/assignment_loss‚1.9715974867921342 1loss/FBt1/assignment_loss‚0.08281791551308676 /loss/FBt2/detection_loss‚0.5457266014923031 epoch‚5 &loss/total_loss‚10.048520727883574AEÐ é „ 2loss/SRqqt1/assignment_loss‚0.6568273057105155 1loss/FBt2/assignment_loss‚0.07760440591814712 /loss/FRt2/detection_loss‚1.3616680835775214 /loss/FBt1/detection_loss‚1.2186744702530319 trainer/global_step‚2249 _runtime‚1922.175911654 _step‚184 0loss/FRt1/assignment_loss‚1.9715974867921342 1loss/FBt1/assignment_loss‚0.08281791551308676 /loss/FBt2/detection_loss‚0.5457266014923031 epoch‚5 &loss/total_loss‚10.048520727883574 0loss/FRt2/assignment_loss‚1.1258186544690814 .loss/FRt1/detection_loss‚0.616098086091726 1loss/SRqqt1/detection_loss‚1.1123864754776445 1loss/SRqqt2/detection_loss‚0.6836387345311264 2loss/SRqqt2/assignment_loss‚0.5956625080572552 # _timestamp‚1.78434226348264e+09¸ÿ¼Þžê ‚(:“ „ÎëÒЂ’” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.697' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.339400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò±{?»ë ‚(:° „ÎëÒϬ‚˜$ disk./.usageGB‚594.3695449829102 disk.dm-0.in‚ 4235.71875! disk.dm-0.out‚6379.8955078125 disk.sda.in‚0 disk.nvme0n1.in‚ 4235.71875 cpu‚26.216185309103597 disk.rbd1.out‚ 4.95703125) disk./.usagePercent‚39.88863887306876 disk.sda1.out‚0 disk.dm-1.out‚ 50.86328125# proc.memory.rssMB‚ 2714.73828125 network.sent‚ 3.425374e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 50.86328125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.56944751739502 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 4235.71875 disk.sda3.in‚0$ disk.nvme0n1.out‚6379.8955078125& disk.nvme0c0n1.out‚6379.8955078125 network.recv‚545909 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 50.86328125 disk.sda2.in‚0$ memory_percent‚95.46904563903809' proc.memory.availableMB‚ 742.3515625 proc.cpu.threads‚17`¼~œì ‚(:‘ “ÎëÒ‹ÙÝ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QD•Æí ‚(:» “ÎëÒ‚€Ê• disk.dm-1.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚39.88867087401054 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 3.475187e+06 disk.dm-0.in‚ 4236.68359375 disk.sda.in‚0 disk.sda.out‚ 51.46484375 disk.sda2.out‚0! disk.nvme0n1.in‚ 4236.68359375) proc.memory.percent‚16.56944751739502 network.recv‚550627 disk.rbd0.in‚0 disk.dm-1.out‚ 51.46484375# disk.nvme0c0n1.in‚ 4236.68359375" disk.dm-0.out‚6396.56982421875' disk.nvme0c0n1.out‚6396.56982421875 disk.sda3.in‚0 disk.sda3.out‚ 51.46484375 cpu‚26.212340250972755 proc.cpu.threads‚17% disk.nvme0n1.out‚6396.56982421875 disk.rbd1.out‚ 4.95703125" memory_percent‚95.469069480896 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 742.34765625# proc.memory.rssMB‚ 2714.73828125$ disk./.usageGB‚594.3700218200684™Á¹î2î - _wandbruntime‚1961  _runtime‚1961š_ ‡Åï ¿ % _timestamp‚1.7843423029764001e+09 _runtime‚ 1961.6700989 _step‚185 & lr-AdamW/pg1‚0.0004387554465978792 & lr-AdamW/pg2‚0.0004387554465978792 trainer/global_step‚2299Âò1Êð Ä _step‚185 & lr-AdamW/pg1‚0.0004387554465978792 & lr-AdamW/pg2‚0.0004387554465978792 trainer/global_step‚2299 % _timestamp‚1.7843423029764001e+09 _runtime‚ 1961.6700989¹±‹Í2ñ - _wandbruntime‚1961  _runtime‚1961ÅÈ÷iÚ'ò ÔO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1961.675689703 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚2299 _step‚186 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2421250096128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348338042388480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920ïIJa©ó ÙO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348971033406858 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2421250096128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348338042388480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1961.675689703 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚2299 _step‚186 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21272245 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4033723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 # _timestamp‚1.78434230359329e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0zûÞ&è&ö âM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2418558718242 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18168422 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349118197543202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348485283418112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4033726 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348485283418112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21272245 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4033723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 # _timestamp‚1.78434230359329e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2499291256320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2499291256320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976¾{Ô Ä2† - _wandbruntime‚2002  _runtime‚2002“ Pyã&‡ ÝM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355847334899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21726697 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚191 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 trainer/global_step‚2349 % _timestamp‚1.7843423438095481e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5034602 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78794532063232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9155578 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5034333 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2473299571200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2002.50343185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2472958224896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4120976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21716706 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304&}‡¬!ˆ âM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355847334899712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9155578 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5034333 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2473299571200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2002.50343185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2472958224896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4120976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch±´sÇ_end/allocation.small_pool.freed‚21716706 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355847334899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21726697 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚191 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 trainer/global_step‚2349 % _timestamp‚1.7843423438095481e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5034602 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78794532063232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18923990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365585127790592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363722975978624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4206478 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80432541231104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5135153 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22172083 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9341355 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776¹  /à'— ÚO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361204195340288 trainer/global_step‚2399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82982884588032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843423831579695e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41086096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18923992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2522982438912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363062128312320 _runtime‚2041.852497158 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82982758524416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18923992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22162106 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5134877 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107451392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22172083 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9341355 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4206481 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80432559843328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2518780638336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18923990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365585127790592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363722975978624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4206478 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80432541231104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5135153 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _runtime‚2042.466268458 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5137196 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5136913 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2523653726720 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 QEDeviceStatsMonitor.on_train_batch_e3ºy¦Knd/inactive_split.large_pool.freed‚4208223 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚195 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361350723051520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18931622 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551336477696 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2523653726720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83017325079552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80465988601856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80465988601856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _runtime‚2042.466268458 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5137196 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5136913 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2523653726720 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4208223 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚195 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361350723051520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18931622 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551336477696 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2523653726720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83017325079552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ã;=ÆT2› - _wandbruntime‚2042  _runtime‚2042BƒÇo œ ƒ trainer/global_step‚2399 _runtime‚2042.468496717 1loss/SRqqt1/detection_loss‚1.0362807846387379 2loss/SRqqt2/assignment_loss‚0.5032508529120946 /loss/FRt2/detection_loss‚1.2950085114543766 /loss/FBt1/detection_loss‚1.1613692172492545 epoch‚5 % _timestamp‚1.7843423837751844e+09 _step‚196 0loss/FRt1/assignment_loss‚2.0344562393530397 2loss/SRqqt1/assignment_loss‚0.7042339530557261 3loss/FBt1/assignment_loss‚0.0068424543642417275 1loss/FBt2/assignment_loss‚0.04296926117493229 /loss/FBt2/detection_loss‚0.6611799187684322 0loss/FRt2/assignment_loss‚0.9885691445408844 /loss/FRt1/detection_loss‚0.5860911892192863 1loss/SRqqt2/detection_loss‚0.6733892967603076 %loss/total_loss‚9.693640823491315©j  ˆ _step‚196 0loss/FRt1/assignment_loss‚2.0344562393530397 2loss/SRqqt1/assignment_loss‚0.7042339530557261 3loss/FBt1/assignment_loss‚0.0068424543642417275 1loss/FBt2/assignment_loss‚0.04296926117493229 /loss/FBt2/detection_loss‚0.6611799187684322 0loss/FRt2/assignment_loss‚0.9885691445408844 /loss/FRt1/detection_loss‚0.5860911892192863 1loss/SRqqt2/detection_loss‚0.6733892967603076 %loss/total_loss‚9.693640823491315 trainer/global_step‚2399 _runtime‚2042.468496717 1loss/SRqqt1/detection_loss‚1.0362807846387379 2loss/SRqqt2/assignment_loss‚0.5032508529120946 /loss/FRt2/detection_loss‚1.2950085114543766 /loss/FBt1/detection_loss‚1.1613692172492545 epoch‚5 % _timestamp‚1.7843423837751844e+09ÄÂÄ‘Ÿžž ‚(:“ üÎëÒ„ÏÌ– gpu.0.gpu‚59.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F½Ÿ ‚(:² üÎëҼ˫˜" disk.dm-0.out‚6452.51904296875 disk.sda.out‚ 53.546875 disk.sda3.in‚0 disk.sda3.out‚ 53.546875 disk.rbd1.in‚0 disk.rbd1.out‚ 5.14453125 disk.rbd0.out‚0) disk./.usagePercent‚39.88870978715575 disk.sda2.out‚0% disk.nvme0n1.out‚6452.51904296875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.in‚0) proc.memory.percent‚16.59562587738037 disk.dm-0.in‚ 4513.33984375! disk.nvme0n1.in‚ 4513.33984375 disk.sda1.in‚0% proc.memory.availableMB‚ 491.53125 network.sent‚ 3.63796e+06 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 4513.33984375# proc.memory.rssMB‚ 2719.02734375 proc.cpu.threads‚17 network.recv‚576488 cpu‚26.21008622043923$ disk./.usageGB‚594.3706016540527 disk.dm-1.in‚0 disk.dm-1.out‚ 53.546875 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚6452.51904296875$ memory_percent‚96.99993133544922¸µhî2  - _wandbruntime‚2059  _runtime‚2059gËf·-¡ Þ{ 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959455132484436 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6494352221488953 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08457290381193161 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8195539116859436 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17152853310108185 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2620936930179596 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7400237917900085 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6194647550582886 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5304166674613953 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7261196970939636 particle/accuracy_3_of_5‚1 _step‚197 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06200814247131348 +jet/accuracy_4_of_4‚0.01628045178949833 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32235628366470337 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07648535072803497 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5607818365097046 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3956778347492218 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864120960235596 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8448601961135864 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2606309652328491 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931044578552246 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7898997068405151 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45785313844680786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8863568305969238 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _runtime‚2059.811685658 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7843424011174016e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5534557104110718 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24112962186336517 -particle/accuracy_4_of_4‚0.94158935546875 +particle/specificity‚0.9026945233345032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6385499238967896 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8918178081512451 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6819433569908142 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7135676741600037 epoch‚5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7588731050491333 *jet/accuracy_1_of_3‚0.9167161583900452 jet/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.20369918644428253 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6840567588806152 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46779295802116394 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9236583709716797 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12858140468597412 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6982709169387817 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2952115535736084 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6258220672607422 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6307346820831299 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05899260565638542 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_3‚0.4880153238773346 )loss/val_total_loss‚9.262507985399777 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765766859054565 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6356363892555237 /particle/accuracy_2_of_2‚0.9998962879180908 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9968975782394409 *jet/accuracy_1_of_2‚0.7883233428001404 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20967067778110504 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_2‚0.1189967542886734 +jet/accuracy_3_of_3‚0.04408945515751839 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19092489778995514 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7693500518798828 particle/accuracy_1_of_1‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19337111711502075 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7459787130355835 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7479508519172668 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2342590093612671 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21697109937667847 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904083013534546 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9896358251571655 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6400310397148132 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0834711417555809 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913808703422546 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6273987293243408 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1278255581855774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 +particle/sensitivity‚0.5931873321533203 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9666851758956909 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6397393345832825 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6713098287582397 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22278177738189697 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 +jet/accuracy_3_of_4‚0.04596220701932907 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9833956360816956 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0771469846367836 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19152182340621948 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972040057182312 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24845150113105774 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9038727879524231 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26416081190109253 particle/accuracy_1_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8195539116859436 6validation_average_jet_accuracy‚0.3709126114845276 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24174806475639343 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2504631578922272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.78385990858078 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.025090396404266357 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3019346594810486 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3800181746482849 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6869268417358398 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6849020719528198 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6374483704566956 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06200814247131348 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959189891815186 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6397393345832825 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 trainer/global_step‚2399 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6379064321517944 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2504631578922272 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8116795420646667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37066614627838135 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33606913685798645 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958223700523376 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031325310468673706 7 Purity/2FRt0SRqqt0FBt/FRt_pug|ü-rity‚0.2227517068386078 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041992027312517166 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04682205244898796 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6122568845748901 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958223700523376 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7395177483558655 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6139203906059265 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7580334544181824 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6865503191947937 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17466190457344055 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7850263714790344 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22216208279132843 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921154975891113 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7371137738227844 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04406134411692619 /particle/accuracy_3_of_4‚0.9914166331291199 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8948603272438049 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09291818737983704 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14519961178302765 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884002208709717 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9590662121772766 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8922713994979858 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084245085716248 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.616092324256897 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25388121604919434 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333432674408 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3848933279514313 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11158930510282516 &particle/accuracy‚0.83623206615448 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795331358909607 'particle/f_score‚0.6087048649787903 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2820685803890228 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21074841916561127 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8638412952423096 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9744628667831421 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2505775988101959 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6463645100593567 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20100827515125275J-!3é=¢ ã{ & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33606913685798645 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9958223700523376 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031325310468673706 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2227517068386078 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041992027312517166 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04682205244898796 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6122568845748901 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9958223700523376 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7395177483558655 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6139203906059265 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7580334544181824 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6865503191947937 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17466190457344055 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7850263714790344 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22216208279132843 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921154975891113 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7371137738227844 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04406134411692619 /particle/accuracy_3_of_4‚0.9914166331291199 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8948603272438049 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09291818737983704 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14519961178302765 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884002208709717 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9590662121772766 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8922713994979858 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084245085716248 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.616092324256897 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25388121604919434 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333432674408 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3848933279514313 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11158930510282516 &particle/accuracy‚0.83623206615448 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795331358909607 'particle/f_score‚0.6087048649787903 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2820685803890228 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21074841916561127 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8638412952423096 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9744628667831421 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2505775988101959 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6463645100593567 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20100827515125275 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959455132484436 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6494352221488953 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08457290381193161 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8195539116859436 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17152853310108185 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2620936930179596 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7400237917900085 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6194647550582886 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5304166674613953 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7261196970939636 particle/accuracy_3_of_5‚1 _step‚197 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06200814247131348 +jet/accuracy_4_of_4‚0.01628045178949833 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32235628366470337 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07648535072803497 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5607818365097046 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3956778347492218 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864120960235596 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8448601961135864 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2606309652328491 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931044578552246 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7898997068405151 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45785313844680786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8863568305969238 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _runtime‚2059.811685658 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7843424011174016e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5534557104110718 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24112962186336517 -particle/accuracy_4_of_4‚0.94158935546875 +particle/specificity‚0.9026945233345032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6385499238967896 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8918178081512451 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6819433569908142 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7135676741600037 epoch‚5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7588731050491333 *jet/accuracy_1_of_3‚0.9167161583900452 jet/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.20369918644428253 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6840567588806152 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46779295802116394 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9236583709716797 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12858140468597412 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6982709169387817 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2952115535736084 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703880310059 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6258220672607422 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6307346820831299 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05899260565638542 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_3‚0.4880153238773346 )loss/val_total_loss‚9.262507985399777 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765766859054565 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6356363892555237 /particle/accuracy_2_of_2‚0.9998962879180908 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9968975782394409 *jet/accuracy_1_of_2‚0.7883233428001404 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20967067778110504 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_2‚0.1189967542886734 +jet/accuracy_3_of_3‚0.04408945515751839 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19092489778995514 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7693500518798828 particle/accuracy_1_of_1‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19337111711502075 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7459787130355835 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7479508519172668 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2342590093612671 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21697109937667847 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904083013534546 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9896358251571655 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6400310397148132 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0834711417555809 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913808703422546 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6273987293243408 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1278255581855774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 +particle/sensitivity‚0.5931873321533203 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9666851758956909 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6397393345832825 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6713098287582397 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22278177738189697 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 +jet/accuracy_3_of_4‚0.04596220701932907 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9833956360816956 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0771469846367836 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19152182340621948 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972040057182312 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24845150113105774 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9038727879524231 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26416081190109253 particle/accuracy_1_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8195539116859436 6validation_average_jet_accuracy‚0.3709126114845276 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24174806475639343 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2504631578922272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.78385990858078 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.025090396404266357 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3019346594810486 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3800181746482849 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6869268417358398 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6849020719528198 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6374483704566956 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06200814247131348 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959189891815186 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6397393345832825 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 trainer/global_step‚2399 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6379064321517944 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2504631578922272 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8116795420646667 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37066614627838135 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258ÅñáäÓž£ ‚(:“ ‹ÏëÒæèÄ– gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡»ÓõŤ ‚(:º ‹ÏëÒµÙì— disk.sda3.in‚0 disk.sda3.out‚ 53.94140625 disk.rbd1.out‚ 5.20703125 disk.rbd0.out‚0 disk.dm-1.out‚ 53.94140625# disk.nvme0c0n1.in‚ 4525.73046875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 53.94140625 disk.rbd1.in‚0 disk.sda.in‚0$ memory_percent‚95.46823501586914$ disk./.usageGB‚594.3714637756348 disk.sda1.in‚0! disk.nvme0n1.in‚ 4525.73046875) disk./.usagePercent‚39.88876764485848 disk.dm-1.in‚0 disk.dm-0.in‚ 4525.73046875" disk.dm-0.out‚6709.11572265625 disk.sda2.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚6709.11572265625 disk.rbd0.in‚0& proc.memory.availableMB‚ 742.484375# proc.memory.rssMB‚ 2719.02734375) proc.memory.percent‚16.59562587738037 cpu‚26.213600664250542 network.recv‚581111' disk.nvme0c0n1.out‚6709.11572265625 network.sent‚ 3.68786e+06rõÉ’¥ ‚(:’ šÏëÒ¾ñ“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÕ¶¦ ‚(:« šÏëÒ³‘ù– network.recv‚585343 disk.sda2.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2719.02734375 disk.sda.in‚0 disk.sda.out‚ 54.1640625 disk.sda1.in‚0 disk.dm-1.out‚ 54.1640625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 742.47265625 disk.sda3.out‚ 54.1640625 disk.dm-0.in‚ 4525.73046875 disk.dm-0.out‚ 6957.421875 cpu‚26.210097234221088 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚95.46830654144287 disk.sda3.in‚0 disk.rbd1.out‚ 5.20703125 network.sent‚ 3.745304e+06$ disk./.usageGB‚594.3707046508789! disk.nvme0n1.in‚ 4525.73046875 disk.nvme0n1.out‚ 6957.421875 disk.rbd1.in‚0" disk.nvme0c0n1.out‚ 6957.421875) disk./.usagePercent‚39.88871669935917 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 4525.73046875) proc.memory.percent‚16.59562587738037ˆg¡©’§ ‚(:‡ ©ÏëÒ°¥Ñ• gpu.0.gpu‚29.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0› ¿º¨ ‚(:¯ ©ÏëÒ’í®— network.sent‚ 3.75225e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 54.421875& proc.memory.availableMB‚ 742.734375) proc.memory.percent‚16.59562587738037 cpu‚26.206609597066034 proc.cpu.threads‚17 network.recv‚588869 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 4552.359375" disk.dm-0.out‚6967.77099609375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4552.359375 disk.sda2.in‚0$ memory_percent‚95.46670913696289 disk.sda.out‚ 54.421875 disk.sda2.out‚0 disk.nvme0n1.in‚ 4552.359375 disk.dm-1.out‚ 54.421875 disk.drbd2165.out‚0% disk.nvme0n1.out‚6967.77099609375 disk.rbd1.out‚ 5.26953125* disk./.usagePercent‚39.888722843539995$ disk./.usageGB‚594.3707962036133 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚6967.77099609375# proc.memory.rssMB‚ 2719.02734375¢Ôiµ2© - _wandbruntime‚2101  _runtime‚2101[²›˜Çª Á % _timestamp‚1.7843424426099317e+09 _runtime‚2101.303482708 _step‚198 & lr-AdamW/pg1‚0.0004370757312210994 & lr-AdamW/pg2‚0.0004370757312210994 trainer/global_step‚2449ð»Ì« Æ _runtime‚2101.303482708 _step‚198 & lr-AdamW/pg1‚0.0004370757312210994 & lr-AdamW/pg2‚0.0004370757312210994 trainer/global_step‚2449 % _timestamp‚1.7843424426099317e+09Æ>MÂj2¬ - _wandbruntime‚2101  _runtime‚2101õ-Cy–­ ÔO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19426258 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22686885 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82119909869568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4296251 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.ono:ö*D_train_batch_start/allocation.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2101.304775484 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22686885 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82119909869568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4296251 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2101.304775484 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚199ÇJ:[ 2¯ - _wandbruntime‚2101  _runtime‚2101®EæRä&° ÞM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372500987473920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372500987473920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42129825 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4297996 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84773885251072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2620546622976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843424432268128e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2592423782912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9572133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2592082436608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695937 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4297993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843424432268128e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2592423782912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9572133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2592082436608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695937 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4297993 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380402016668204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4383501 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 trainer/global_step‚2499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248e›mÁÞ'¾ ØO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380402016668204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4383501 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 trainer/global_step‚2499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438ˈ'qÎ2¿ - _wandbruntime‚2141  _runtime‚2141v#›…À ÞM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638744391108 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379863038955520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4385243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2642777938432 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 trainer/global_step‚2499 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19815490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23150389 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_byteœµ‡ñ_s.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83824776554496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843424831800401e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚2141.873932215 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Q)ûé&Á ãM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19815490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23150389 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83824776554496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843424831800401e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚2141.873932215 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638744391108 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379863038955520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4385243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2642777938432 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 trainer/global_step‚2499 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304ÌûÊàÍ2 - _wandbruntime‚2141  _runtime‚2141ÁF1åà ‚ trainer/global_step‚2499 /loss/FBt2/detection_loss‚0.5668100335804577 2loss/SRqqt1/assignment_loss‚0.6337953195726814 1loss/FBt2/assignment_loss‚0.08965528772219365 /loss/FRt2/detection_loss‚1.3217414619145034 _step‚205 2loss/SRqqt2/assignment_loss‚0.6485271494141939 % _timestamp‚1.7843424831823022e+09 _runtime‚2141.875688537 &loss/total_loss‚10.131724586982042 0loss/FRt1/assignment_loss‚2.0291983161848117 /loss/FRt1/detection_loss‚0.5571046773154394 1loss/SRqqt1/detection_loss‚1.0568513206180028 1loss/SRqqt2/detection_loss‚0.7229230958523691 /loss/FBt1/detection_loss‚1.1150875996911165 0loss/FRt2/assignment_loss‚1.2849094240022567 1loss/FBt1/assignment_loss‚0.10512090111401565 epoch‚6°¶Ò, Ä ‡ % _timestamp‚1.7843424831823022e+09 _runtime‚2141.875688537 &loss/total_loss‚10.131724586982042 0loss/FRt1/assignment_loss‚2.0291983161848117 /loss/FRt1/detection_loss‚0.5571046773154394 1loss/SRqqt1/detection_loss‚1.0568513206180028 1loss/SRqqt2/detection_loss‚0.7229230958523691 /loss/FBt1/detection_loss‚1.1150875996911165 0loss/FRt2/assignment_loss‚1.2849094240022567 1loss/FBt1/assignment_loss‚0.10512090111401565 epoch‚6 trainer/global_step‚2499 /loss/FBt2/detection_loss‚0.5668100335804577 2loss/SRqqt1/assignment_loss‚0.6337953195726814 1loss/FBt2/assignment_loss‚0.08965528772219365 /loss/FRt2/detection_loss‚1.3217414619145034 _step‚205 2loss/SRqqt2/assignment_loss‚0.6485271494141939Í­]kq“Å ‚(:ˆ ÖÏëÒŒýÁ“ gpu.0.gpu‚95.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚243.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.7568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã]…ÆÆ ‚(:» ÖÏëÒžÔ– disk.dm-1.out‚ 56.9296875$ memory_percent‚95.47102451324463 cpu‚26.19625474069785% disk.nvme0n1.out‚7003.11181640625" disk.dm-0.out‚7003.11181640625 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.592860221862793$ disk./.usageGB‚594.3749542236328 disk.sda3.out‚ 56.9296875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4645.37890625 proc.cpu.threads‚17 network.recv‚600473! disk.nvme0n1.in‚ 4645.37890625' disk.nvme0c0n1.out‚7003.11181640625( proc.memory.availableMB‚ 742.02734375* disk./.usagePercent‚39.889001891752315 disk.rbd1.out‚ 5.39453125# proc.memory.rssMB‚ 2718.57421875 disk.sda.out‚ 56.9296875 network.sent‚ 3.823222e+06 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4645.37890625 disk.sda2.in‚0ÜÉ«ÖžÇ ‚(:“ åÏëÒ·Íד gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n ¾È ‚(:³ åÏëҳᕠdisk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚95.47386169433594 network.recv‚605008$ disk./.usageGB‚594.3747978210449# proc.memory.rssMB‚ 2719.02734375 proc.cpu.threads‚17 disk.sda.out‚ 57.61328125) proc.memory.percent‚16.59562587738037 cpu‚26.19310470327912* disk./.usagePercent‚39.888991395443405 disk.sda1.out‚0! disk.dm-0.out‚7011.9462890625 disk.sda.in‚0 disk.dm-0.in‚ 4645.7734375 disk.rbd1.in‚0 disk.sda3.out‚ 57.61328125$ proc.memory.availableMB‚741.5625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 4645.7734375 network.sent‚ 3.886066e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 57.61328125 disk.sda2.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚7011.9462890625 disk.sda2.out‚0$ disk.nvme0n1.out‚7011.9462890625 disk.rbd1.out‚ 5.45703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 4645.7734375sšS²É ‚(:’ ôÏëÒŠ«Ñ“ gpu.0.gpu‚51.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÄëHÇÊ ‚(:¼ ôÏëÒ‹‘À•' disk.nvme0c0n1.out‚7019.02294921875 disk.sda.out‚ 57.91015625 disk.sda2.out‚0! disk.nvme0n1.in‚ 4910.50390625 disk.rbd1.in‚0( proc.memory.availableMB‚ 741.55859375# proc.memory.rssMB‚ 2719.02734375 cpu‚26.18975460804247) disk./.usagePercent‚39.88894787416259 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚608240% disk.nvme0n1.out‚7019.02294921875 disk.rbd1.out‚ 5.45703125) proc.memory.percent‚16.59562587738037 disk.dm-1.out‚ 57.91015625" disk.dm-0.out‚7019.02294921875 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 4910.50390625 disk.sda3.in‚0$ disk./.usageGB‚594.3741493225098 disk.sda1.in‚0 disk.dm-0.in‚ 4910.50390625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 57.91015625 network.sent‚ 3.891141e+06 disk.dm-1.in‚0$ memory_percent‚95.473885536193857Êáx2Ë - _wandbruntime‚2181  _runtime‚21810ÀNÃÇÌ Á _runtime‚2181.325342348 _step‚206 & lr-AdamW/pg1‚0.0004358876066749403 & lr-AdamW/pg2‚0.0004358876066749403 trainer/global_step‚2549 % _timestamp‚1.7843425226317997e+09C ÌÍ Æ _runtime‚2181.325342348 _step‚206 & lr-AdamW/pg1‚0.0004358876066749403 & lr-AdamW/pg2‚0.0004358876066749403 trainer/global_step‚2549 % _timestamp‚1.7843425226317997e+09ΫSJ§2Î - _wandbruntime‚2181  _runtime‚2181ÑÌ0Ú'Ï ÔO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚2181.326697389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23605766 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23595789 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385089476374528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692460813312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚2549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389770310220800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385089459335168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387077849407488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85462785722368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387077849407488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20189458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9950977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389770668599296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20189458 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 % _timestamp‚1.7843425226328428e+09)o€vÐ ÙO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚2181.326697389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23605766 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23595789 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385089476374528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692460813312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚2549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389770310220800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385089459335168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split‰ÊZöÏ.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387077849407488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85462785722368 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387077849407488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20189458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9950977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389770668599296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20189458 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 % _timestamp‚1.7843425226328428e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258Ï3#x„2Ñ - _wandbruntime‚2181  _runtime‚2181_HRtä&Ò ÞM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9955068 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2693473440256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2693473440256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2689360878016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843425232481678e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387225107476480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2693814786560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2549 _step‚208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚2181.942035635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0X²ié&Ó ãM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2693473440256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2689360878016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843425232481678e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387225107476480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2693814786560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2549 _step‚208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚2181.942035635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9955068 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2693473440256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929ЯºÓ[2Ô - _wandbruntime‚2181  _runtime‚2181¬?å*Õ € /loss/FRt2/detection_loss‚1.3190736853757994 1loss/SRqqt1/detection_loss‚0.9613725019306386 &loss/total_loss‚10.284006976289508 epoch‚6 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚0.6821759657963603 0loss/FBt1/assignment_loss‚0.1390533335493057 /loss/FBt1/detection_loss‚1.7952749696338461 /loss/FBt2/detection_loss‚0.5142156635460412 % _timestamp‚1.7843425232512777e+09 _runtime‚2181.944990335 0loss/FRt2/assignment_loss‚1.1148173175449294 2loss/SRqqt2/assignment_loss‚0.5345606473395047 1loss/SRqqt2/detection_loss‚0.7049264636741358 _step‚209 0loss/FRt1/assignment_loss‚1.8878272095786743 0loss/FBt2/assignment_loss‚0.0698258292465427 /loss/FRt1/detection_loss‚0.5608833890737291f]Xá Ö … % _timestamp‚1.7843425232512777e+09 _runtime‚2181.944990335 0loss/FRt2/assignment_loss‚1.1148173175449294 2loss/SRqqt2/assignment_loss‚0.5345606473395047 1loss/SRqqt2/detection_loss‚0.7049264636741358 _step‚209 0loss/FRt1/assignment_loss‚1.8878272095786743 0loss/FBt2/assignment_loss‚0.0698258292465427 /loss/FRt1/detection_loss‚0.5608833890737291 /loss/FRt2/detection_loss‚1.3190736853757994 1loss/SRqqt1/detection_loss‚0.9613725019306386 &loss/total_loss‚10.284006976289508 epoch‚6 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚0.6821759657963603 0loss/FBt1/assignment_loss‚0.1390533335493057 /loss/FBt1/detection_loss‚1.7952749696338461 /loss/FBt2/detection_loss‚0.5142156635460412ÑqÌ–4“× ‚(:ˆ ƒÐëÒÑÌç– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚241.853' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<»»Ø ‚(:° ƒÐëÒ²§ˆ™ disk.sda3.in‚0# proc.memory.rssMB‚ 2719.02734375! disk.dm-0.out‚7028.9423828125 disk.nvme0n1.in‚ 4911.46875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚611677 disk.nvme0c0n1.in‚ 4911.46875& disk.nvme0c0n1.out‚7028.9423828125 disk.rbd1.out‚ 5.51953125 cpu‚26.18658119756172 network.sent‚ 3.909121e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 4911.46875 disk.sda3.out‚ 58.30859375) proc.memory.percent‚16.59562587738037 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.out‚ 58.30859375 disk.rbd1.in‚0$ memory_percent‚95.47257423400879' proc.memory.availableMB‚ 741.7734375 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚594.3743209838867 disk.dm-1.out‚ 58.30859375 disk.drbd2165.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚7028.9423828125 disk.rbd0.in‚0* disk./.usagePercent‚39.888959394501626(OWR“Ù ‚(:ˆ ’ÐëÒߪÀ“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚235.669' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä ó ¾Ú ‚(:³ ’ÐëÒ‡°– disk.nvme0n1.in‚ 4911.46875 disk.sda3.out‚ 58.58984375 disk.dm-0.in‚ 4911.46875" memory_percent‚95.469069480896# proc.memory.rssMB‚ 2718.57421875 disk.rbd1.out‚ 5.51953125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 4911.46875 disk.sda.in‚0* proc.memory.percent‚16.592860221862793 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7032.35986328125" disk.dm-0.out‚7032.35986328125) disk./.usagePercent‚39.88895401834341 disk.dm-1.out‚ 58.58984375 network.sent‚ 3.958987e+06$ disk./.usageGB‚594.3742408752441 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚616010% disk.nvme0n1.out‚7032.35986328125 disk.sda3.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 742.34765625 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.183268146708684 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 58.58984375 disk.sda2.out‚0gA9œÛ ‚(:‘ ¡ÐëÒÈî• gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RZ²ÃÜ ‚(:¸ ¡ÐëÒàij–% disk.nvme0n1.out‚7042.17431640625 network.recv‚619300# disk.nvme0c0n1.in‚ 4912.30078125" disk.dm-0.out‚7042.17431640625 disk.sda2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 3.964013e+06 disk.rbd1.in‚0! disk.nvme0n1.in‚ 4912.30078125 disk.drbd2165.out‚0) proc.memory.percent‚16.58952236175537' disk.nvme0c0n1.out‚7042.17431640625 disk.sda.out‚ 58.8515625 cpu‚26.18015466013021 disk.sda.in‚0 disk.sda1.in‚0$ disk./.usageGB‚594.3742065429688 disk.sda3.out‚ 58.8515625 disk.sda1.out‚0 disk.dm-1.out‚ 58.8515625 disk.drbd2165.in‚0 disk.dm-0.in‚ 4912.30078125 disk.sda2.out‚0 disk.rbd1.out‚ 5.58203125 disk.rbd0.in‚0( disk./.usagePercent‚39.8889517142756 disk.sda3.in‚0$ memory_percent‚95.46759128570557( proc.memory.availableMB‚ 742.58984375# proc.memory.rssMB‚ 2718.02734375 disk.dm-1.in‚0: ‹u2Ý - _wandbruntime‚2221  _runtime‚2221×CY6ÇÞ Á & lr-AdamW/pg1‚0.0004352731632654813 & lr-AdamW/pg2‚0.0004352731632654813 trainer/global_step‚2599 % _timestamp‚1.7843425624920862e+09 _runtime‚2221.185795684 _step‚210´TžÌß Æ _runtime‚2221.185795684 _step‚210 & lr-AdamW/pg1‚0.0004352731632654813 & lr-AdamW/pg2‚0.0004352731632654813 trainer/global_step‚2599 % _timestamp‚1.7843425624920862e+09Ò~vy2à - _wandbruntime‚2221  _runtime‚2221ã'l‰|á ÖO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batc™«ç`&h_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394439917928448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20571056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2743156315136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4557998 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397183415582720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394439900889088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24060218 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392415009968128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843425624933527e+09) TÌá'â ÛO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397183415582720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394439900889088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24060218 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392415009968128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843425624933527e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394439917928448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20571056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2743156315136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4557998 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Ó êï32ã - _wandbruntime‚2221  _runtime‚2221PÔ÷mæ&ä àM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20578688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10146019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89942026853888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5586273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392561503600640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2739977364924 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394587158958080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392561503600640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2739977364924 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394587158958080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10146019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89942026853888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5586273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360Ô–sDì2æ - _wandbruntime‚2221  _runtime‚2221}¯tÑç € _runtime‚2221.805590837 _step‚213 1loss/SRqqt1/assignment_loss‚0.747695626111063 2loss/SRqqt2/assignment_loss‚0.6150023377277645 1loss/FBt1/assignment_loss‚0.15144870795926552 1loss/SRqqt1/detection_loss‚1.0330055476465252 trainer/global_step‚2599 1loss/SRqqt2/detection_loss‚0.7045583500096452 %loss/total_loss‚9.696802382171779 epoch‚6 % _timestamp‚1.7843425631123054e+09 2loss/FBt2/assignment_loss‚0.048806193414021974 0loss/FRt1/assignment_loss‚1.8807464241348089 0loss/FRt2/assignment_loss‚1.1412814226667238 .loss/FRt1/detection_loss‚0.589734184952972 /loss/FRt2/detection_loss‚1.2762329783671298 /loss/FBt1/detection_loss‚0.9615079066250474 /loss/FBt2/detection_loss‚0.5467827025568113Ö°%@ è … 0loss/FRt1/assignment_loss‚1.8807464241348089 0loss/FRt2/assignment_loss‚1.1412814226667238 .loss/FRt1/detection_loss‚0.589734184952972 /loss/FRt2/detection_loss‚1.2762329783671298 /loss/FBt1/detection_loss‚0.9615079066250474 /loss/FBt2/detection_loss‚0.5467827025568113 _runtime‚2221.805590837 _step‚213 1loss/SRqqt1/assignment_loss‚0.747695626111063 2loss/SRqqt2/assignment_loss‚0.6150023377277645 1loss/FBt1/assignment_loss‚0.15144870795926552 1loss/SRqqt1/detection_loss‚1.0330055476465252 trainer/global_step‚2599 1loss/SRqqt2/detection_loss‚0.7045583500096452 %loss/total_loss‚9.696802382171779 epoch‚6 % _timestamp‚1.7843425631123054e+09 2loss/FBt2/assignment_loss‚0.048806193414021974ÕÙÐÄß’é ‚(:‡ °ÐëÒîϧ“ gpu.0.gpu‚26.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýˆ¬ö¿ê ‚(:´ °ÐëÒååã— disk.sda1.in‚0! disk.nvme0c0n1.in‚ 5021.765625 disk.dm-0.in‚ 5021.765625 disk.sda.in‚0 disk.nvme0n1.in‚ 5021.765625 cpu‚26.177160736036676% disk.nvme0n1.out‚7049.66357421875 disk.sda3.out‚ 59.0703125 proc.cpu.threads‚17 network.sent‚ 3.96883e+06 network.recv‚622691 disk.rbd1.out‚ 5.58203125# proc.memory.rssMB‚ 2718.57421875* proc.memory.percent‚16.592860221862793* disk./.usagePercent‚39.888955298381084 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚7049.66357421875 disk.sda2.out‚0$ disk./.usageGB‚594.3742599487305 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 59.0703125 disk.drbd2165.out‚0 disk.rbd1.in‚0$ memory_percent‚95.47228813171387' proc.memory.availableMB‚ 741.8203125 disk.drbd2165.in‚0" disk.dm-0.out‚7049.66357421875 disk.sda.out‚ 59.070312528¶“ë ‚(:ˆ ¿ÐëÒ–³é“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¤Vøºì ‚(:¯ ¿ÐëÒè – disk.dm-0.in‚ 5021.765625# proc.memory.rssMB‚ 2719.02734375 proc.cpu.threads‚17 network.sent‚ 4.031841e+06 disk.rbd1.out‚ 5.64453125) proc.memory.percent‚16.59562587738037 disk.rbd1.in‚0$ memory_percent‚95.47457695007324 cpu‚26.17384276575194$ disk./.usageGB‚594.3742294311523 disk.dm-1.in‚0 disk.sda3.out‚ 59.4765625 disk.drbd2165.out‚0 disk.sda.in‚0 network.recv‚627242 disk.rbd0.out‚0& disk.nvme0c0n1.out‚7054.8740234375 disk.sda3.in‚0' proc.memory.availableMB‚ 741.4453125) disk./.usagePercent‚39.88895325032081 disk.rbd0.in‚0 disk.dm-1.out‚ 59.4765625! disk.nvme0c0n1.in‚ 5021.765625! disk.dm-0.out‚7054.8740234375 disk.sda.out‚ 59.4765625 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5021.765625$ disk.nvme0n1.out‚7054.8740234375 disk.sda2.out‚0 disk.sda1.in‚0C¨¯q2í - _wandbruntime‚2261  _runtime‚2261±Ú1Åî ¿ trainer/global_step‚2649 % _timestamp‚1.7843426026534548e+09 _runtime‚2261.346979302 _step‚214 % lr-AdamW/pg1‚0.000434645184760549 % lr-AdamW/pg2‚0.000434645184760549=(—Êï Ä trainer/global_step‚2649 % _timestamp‚1.7843426026534548e+09 _runtime‚2261.346979302 _step‚214 % lr-AdamW/pg1‚0.000434645184760549 % lr-AdamW/pg2‚0.000434645184760549Ö¼€\—2ð - _wandbruntime‚2261  _runtime‚2261sž–(Ú'ñ ÔO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24514670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824826462720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24504693 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚2649 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2789582798110 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784342602654487e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2794193156096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20952656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚215 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚215 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2789582798110 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784342602654487e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2794193156096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0×È÷ÅÝ2ó - _wandbruntime‚2261  _runtime‚22611rŸƒjô àM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDevilœË/|&ceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2825759602688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20960288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24523736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5689808 _step‚216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4646993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404744416229888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2795205790208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401949210439680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402677988635434 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2794864443904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2649 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91664942700544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745iØ“™ë&õ åM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4646993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404744416229888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2795205790208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401949210439680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402677988635434 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2794864443904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2649 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91664942700544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2825759602688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20960288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24523736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5689808 _step‚216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993ØTnn2ö - _wandbruntime‚2261  _runtime‚2261ôœ³÷ € 0loss/FRt1/assignment_loss‚1.9346912304560344 2loss/SRqqt1/assignment_loss‚0.6446659685727468 /loss/FBt1/detection_loss‚1.1294818649237806 /loss/FBt2/detection_loss‚0.6015040124705258 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚1.0797985502817098 2loss/FBt2/assignment_loss‚0.025104500238283384 /loss/FRt1/detection_loss‚0.6165572204731539 1loss/SRqqt2/detection_loss‚0.7242945273963085 _runtime‚2261.966187726 1loss/FBt1/assignment_loss‚0.06887561990349798 0loss/SRqqt1/detection_loss‚0.991703777233275 _step‚217 2loss/SRqqt2/assignment_loss‚0.5701045375834024 .loss/FRt2/detection_loss‚1.290272285571114 %loss/total_loss‚9.677054095103832 epoch‚6 % _timestamp‚1.7843426032725358e+09N-ºÎ ø … .loss/FRt2/detection_loss‚1.290272285571114 %loss/total_loss‚9.677054095103832 epoch‚6 % _timestamp‚1.7843426032725358e+09 0loss/FRt1/assignment_loss‚1.9346912304560344 2loss/SRqqt1/assignment_loss‚0.6446659685727468 /loss/FBt1/detection_loss‚1.1294818649237806 /loss/FBt2/detection_loss‚0.6015040124705258 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚1.0797985502817098 2loss/FBt2/assignment_loss‚0.025104500238283384 /loss/FRt1/detection_loss‚0.6165572204731539 1loss/SRqqt2/detection_loss‚0.7242945273963085 _runtime‚2261.966187726 1loss/FBt1/assignment_loss‚0.06887561990349798 0loss/SRqqt1/detection_loss‚0.991703777233275 _step‚217 2loss/SRqqt2/assignment_loss‚0.5701045375834024ÙЬV“ù ‚(:ˆ ÎÐëÒ°úÕ” gpu.0.gpu‚65.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.&½¢Ãú ‚(:¸ ÎÐëÒãî·•& proc.memory.availableMB‚ 741.453125 disk.dm-0.in‚ 5022.59765625 disk.sda2.in‚0! disk.nvme0n1.in‚ 5022.59765625 disk.rbd1.out‚ 5.64453125# proc.memory.rssMB‚ 2719.02734375 disk.sda.in‚0 disk.sda3.in‚0 cpu‚26.170866617903982 network.recv‚630408 disk.dm-1.out‚ 59.6796875 network.sent‚ 4.036766e+06 disk.drbd2165.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.59562587738037$ disk./.usageGB‚594.3742370605469 disk.sda.out‚ 59.6796875 disk.sda3.out‚ 59.6796875 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.88895376233587 disk.rbd0.in‚0$ memory_percent‚95.47452926635742 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 5022.59765625' disk.nvme0c0n1.out‚7062.21728515625" disk.dm-0.out‚7062.21728515625% disk.nvme0n1.out‚7062.21728515625èô(“û ‚(:ˆ ÝÐëÒ½ÓØ“ gpu.0.gpu‚65.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚252.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e‡ó©¹ü ‚(:® ÝÐëÒè´×— disk.dm-0.in‚ 5023.5625! disk.dm-0.out‚7072.3056640625) proc.memory.percent‚16.59562587738037 cpu‚26.168079185788557 network.sent‚ 4.099611e+06 network.recv‚634948 disk.sda3.in‚0& disk.nvme0c0n1.out‚7072.3056640625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 59.94140625# proc.memory.rssMB‚ 2719.02734375$ disk.nvme0n1.out‚7072.3056640625 disk.sda3.out‚ 59.94140625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 5023.5625 disk.sda2.out‚0$ disk./.usageGB‚594.3741188049316 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚96.97718620300293 disk.drbd2165.in‚0' proc.memory.availableMB‚ 495.2578125 proc.cpu.threads‚17* disk./.usagePercent‚39.888945826102315 disk.nvme0n1.in‚ 5023.5625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 59.94140625 disk.rbd1.out‚ 5.76953125T!óžý ‚(:“ ìÐëÒ§·¤“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_b8L½þ ‚(:² ìÐëÒžÑä• disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚96.97713851928711& proc.memory.availableMB‚ 495.265625) proc.memory.percent‚16.59562587738037 disk.sda2.out‚0% disk.nvme0n1.out‚7074.95166015625 network.sent‚ 4.10468e+06* disk./.usagePercent‚39.888965026667385 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 5322.1328125# proc.memory.rssMB‚ 2719.02734375 disk.rbd0.out‚0 disk.sda.out‚ 60.171875 disk.sda3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚594.3744049072266 disk.dm-1.out‚ 60.171875' disk.nvme0c0n1.out‚7074.95166015625 cpu‚26.164994670699325 disk.nvme0n1.in‚ 5322.1328125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 5322.1328125" disk.dm-0.out‚7074.95166015625 network.recv‚638442 disk.sda.in‚0 disk.sda3.out‚ 60.171875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 5.76953125î€\š2ÿ - _wandbruntime‚2301  _runtime‚2301¥v,É€ à ' lr-AdamW/pg1‚0.00043400371149402946 ' lr-AdamW/pg2‚0.00043400371149402946 trainer/global_step‚2699 % _timestamp‚1.7843426428319087e+09 _runtime‚2301.525545443 _step‚218ËéœSÎ È _runtime‚2301.525545443 _step‚218 ' lr-AdamW/pg1‚0.00043400371149402946 ' lr-AdamW/pg2‚0.00043400371149402946 trainer/global_step‚2699 % _timestamp‚1.7843426428319087e+09Ú‰ëØ)2‚ - _wandbruntime‚2301  _runtime‚2301¨îƒ ÖO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409906259400986 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409164003852288 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4732498 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409164020891648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843426428330429e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24969122 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2840539834908 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚2301.530355257 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409164003852288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5791254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pogó î ol.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚2301.530355257 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409164003852288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5791254 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24968197 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90510528260096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2841550888742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4734246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2877067443200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10527816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21341888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845559945728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24978188 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2876978873344 _step‚220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845559945728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24978188 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2876978873344 _step‚220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24968197 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90510528260096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2841550888742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4734246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.oÇVèõn_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2877067443200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10527816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21341888Ü€Â^{2ˆ - _wandbruntime‚2302  _runtime‚2302QÕ=‰ ý _step‚221 1loss/FBt2/assignment_loss‚0.01671548894231763 /loss/FRt2/detection_loss‚1.2474237886196562 1loss/SRqqt1/detection_loss‚0.9948340546776308 % _timestamp‚1.7843426434506748e+09 0loss/FRt1/assignment_loss‚1.8806407238888816 1loss/FBt1/assignment_loss‚0.05971554865482543 /loss/FBt1/detection_loss‚0.9149658127177146 _runtime‚2302.143951017 /loss/FRt1/detection_loss‚0.5959044260965277 %loss/total_loss‚9.237910099371598 epoch‚6 /loss/FRt2/assignment_loss‚1.059217673175189 2loss/SRqqt1/assignment_loss‚0.6859172857091835 2loss/SRqqt2/assignment_loss‚0.5589157261076542 /loss/SRqqt2/detection_loss‚0.67290804268711 .loss/FBt2/detection_loss‚0.550751528094908 trainer/global_step‚2699—ZŽåŠ ‚ 0loss/FRt1/assignment_loss‚1.8806407238888816 1loss/FBt1/assignment_loss‚0.05971554865482543 /loss/FBt1/detection_loss‚0.9149658127177146 _runtime‚2302.143951017 /loss/FRt1/detection_loss‚0.5959044260965277 %loss/total_loss‚9.237910099371598 epoch‚6 /loss/FRt2/assignment_loss‚1.059217673175189 2loss/SRqqt1/assignment_loss‚0.6859172857091835 2loss/SRqqt2/assignment_loss‚0.5589157261076542 /loss/SRqqt2/detection_loss‚0.67290804268711 .loss/FBt2/detection_loss‚0.550751528094908 trainer/global_step‚2699 _step‚221 1loss/FBt2/assignment_loss‚0.01671548894231763 /loss/FRt2/detection_loss‚1.2474237886196562 1loss/SRqqt1/detection_loss‚0.9948340546776308 % _timestamp‚1.7843426434506748e+09ÝS§0“‹ ‚(:ˆ ûÐëÒì†Ü“ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á{ÝÄŒ ‚(:¹ ûÐëÒ¤Ÿü• disk.drbd2165.out‚0( proc.memory.availableMB‚ 495.21484375 cpu‚26.162114606172825* disk./.usagePercent‚39.888991907458475 disk.sda.out‚ 60.3984375% disk.nvme0n1.out‚7086.71923828125 disk.sda3.out‚ 60.3984375 disk.rbd0.out‚0 network.sent‚ 4.1096e+06 disk.dm-0.in‚ 5322.96484375# disk.nvme0c0n1.in‚ 5322.96484375) proc.memory.percent‚16.59562587738037 disk.sda1.in‚0 disk.dm-1.out‚ 60.3984375 disk.drbd2165.in‚0" disk.dm-0.out‚7086.71923828125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2719.02734375' disk.nvme0c0n1.out‚7086.71923828125 proc.cpu.threads‚17 network.recv‚641751$ disk./.usageGB‚594.3748054504395$ memory_percent‚96.97744846343994 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 5.83203125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 5322.96484375 disk.rbd0.in‚0 disk.dm-1.in‚0"2¾§“ ‚(:ˆ ŠÑëÒŸ±Ì“ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚250.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é1ÅŽ ‚(:º ŠÑëÒÓщ— disk.dm-1.in‚0 disk.dm-1.out‚ 60.72265625" disk.dm-0.out‚7094.38916015625 disk.nvme0n1.in‚ 5323.9296875 disk.rbd1.in‚0 network.recv‚646218 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚26.15935652721927 disk.sda1.in‚0 disk.rbd1.out‚ 5.83203125 proc.cpu.threads‚17 disk.sda3.in‚0* disk./.usagePercent‚39.888996003579024 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5323.9296875 disk.sda.out‚ 60.72265625 disk.drbd2165.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚7094.38916015625$ memory_percent‚96.97706699371338 network.sent‚ 4.172463e+06$ disk./.usageGB‚594.3748664855957( proc.memory.availableMB‚ 495.27734375# proc.memory.rssMB‚ 2719.02734375) proc.memory.percent‚16.59562587738037 disk.sda3.out‚ 60.72265625' disk.nvme0c0n1.out‚7094.38916015625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 5323.9296875 disk.sda.in‚0¥1sr“ ‚(:ˆ ™ÑëÒñÛ®“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08_ÀB¸ ‚(:­ ™ÑëÒ¡þÓ• disk.sda2.out‚0 disk.nvme0n1.in‚ 5323.9296875 disk.rbd0.in‚0) proc.memory.percent‚16.59259796142578 disk.dm-0.in‚ 5323.9296875 disk.sda3.out‚ 61.12890625 proc.memory.rssMB‚ 2718.53125 disk.dm-1.out‚ 61.12890625 disk.dm-0.out‚ 7100.34765625 disk.rbd0.out‚0 cpu‚26.15648027370736$ disk.nvme0c0n1.out‚ 7100.34765625 disk.rbd1.out‚ 5.89453125' proc.memory.availableMB‚ 495.7734375 proc.cpu.threads‚17 network.recv‚649776" disk.nvme0c0n1.in‚ 5323.9296875 disk.dm-1.in‚0$ memory_percent‚96.97403907775879 network.sent‚ 4.177436e+06* disk./.usagePercent‚39.888986531300255 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 61.12890625 disk.sda2.in‚0" disk.nvme0n1.out‚ 7100.34765625 disk.sda3.in‚0 disk.sda1.out‚0$ disk./.usageGB‚594.3747253417969?k2‘ - _wandbruntime‚2341  _runtime‚2341bC:Å’ ¿ & lr-AdamW/pg1‚0.0004333487846665518 & lr-AdamW/pg2‚0.0004333487846665518 trainer/global_step‚2749 $ _timestamp‚1.784342682661528e+09 _runtime‚ 2341.35513332 _step‚222–½YÊ“ Ä & lr-AdamW/pg1‚0.0004333487846665518 & lr-AdamW/pg2‚0.0004333487846665518 trainer/global_step‚2749 $ _timestamp‚1.784342682661528e+09 _runtime‚ 2341.35513332 _step‚222Þ£ÀR2” - _wandbruntime‚2341  _runtime‚2341R¡¡kÛ'• ÕO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92148518815744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2895584159744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21715858 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2891156321816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25413597 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419421656532992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10713570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2341.356569051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4819748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927004111872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2895242820608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92148537427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25423574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419421298154496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95075522927616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414391593709568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416526055333888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416526072373248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417282409481494F^ºWà'– ÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95075636408320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10713830 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2895242820608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2341.356569051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4819748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927004111872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2895242820608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92148537427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25423574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419421298154496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95075522927616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414391593709568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416526055333888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416526072373248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417282409481494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92148518815744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2895584159744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21715858 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2891156321816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25413597 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419421656532992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10713570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95110087194112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚2341.971608617 _step‚224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92181947574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚§èo4821496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95109980012032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2928032437760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784342683277786e+093äÜñê&™ äM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417429931207088 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95109980012032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2928032437760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784342683277786e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21723488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95110087194112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚2341.971608617 _step‚224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92181947574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4821496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878026 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚2381.417256471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2941772808724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843427227234507e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22097458 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423888123854848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423888106815488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2799 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÖC” ¦ ÛO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚2381.417256471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2941772808724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843427227234507e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22097458 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423888123854848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423888106815488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2799 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96798261623296 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22097456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878026 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144ã SöÇ2§ - _wandbruntime‚2382  _runtime‚2382ÚS‡å&¨ ßM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421863655014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784342723338497e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22105090 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚228 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25887092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144+Qê&© äM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25887092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421863655014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784342723338497e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22105090 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚228 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.837154746055603 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2232438027858734 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9326568841934204 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6273972988128662 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 particle/accuracy_2_of_2‚1 8 PurityÔ-ÂM5/2FRt1SRqqt0FBt/FRt_purity‚0.19749654829502106 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6024418473243713 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 *jet/accuracy_1_of_2‚0.7761469483375549 jet/accuracy_1_of_5‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49559611082077026 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20778630673885345 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6512986421585083 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7070336937904358 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7471150755882263 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6901841163635254 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4385921061038971 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/sensitivity‚0.6013011336326599 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2707775831222534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2193213701248169 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7375200390815735 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 *jet/accuracy_1_of_3‚0.9122800230979919 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.27658164501190186 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6957243084907532 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922221899032593 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6925548911094666 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6555728912353516 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7496275901794434 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8388512134552002 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946234822273254 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3513830602169037 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7553712129592896 +jet/accuracy_1_of_1‚0.32302480936050415 7validation_average_jet_accuracy‚0.36203211545944214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.05483259633183479 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7318395972251892 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020174413919448853 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18654386699199677 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6582232713699341 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7304043769836426 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚230 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934182167053223 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3269789218902588 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04563792794942856 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.837154746055603 % _timestamp‚1.7843427404739008e+09 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039606425911188126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16234450042247772 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21162793040275574 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8421921730041504 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3817979693412781 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04202000051736832 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6453130841255188 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17417219281196594 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2766585946083069 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5803357362747192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1877547651529312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 'particle/f_score‚0.6206983327865601 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4912283718585968 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0430738590657711 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6418932676315308 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 +jet/accuracy_4_of_4‚0.01628045178949833 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7673410773277283 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6422370076179504 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2244926393032074 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18671837449073792 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1304996758699417 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213908672332764 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18321841955184937 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8084079027175903 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878384470939636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21162793040275574 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4191724359989166 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3083656430244446 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_3_of_5‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02344956435263157 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09844423830509186 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937272071838379 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1662534475326538 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1733028143644333 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962902069091797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 )loss/val_total_loss‚8.980026540261347 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3759167790412903 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.665765106678009 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.038429636508226395 *jet/accuracy_2_of_3‚0.5085383653640747 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20350582897663116 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953615069389343 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6443200707435608 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16890081763267517 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18662221729755402 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.799622118473053 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 *jet/accuracy_2_of_4‚0.7349887490272522 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955456256866455 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7991392016410828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8503969311714172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6207932829856873 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.829176127910614 .particle/accuracy_3_of_4‚0.997619092464447 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8203892707824707 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012263154610991478 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.038429636508226395 *particle/specificity‚0.908067524433136 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18737079203128815 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5917336344718933 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚2799 _runtime‚2399.168275124 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11725087463855743 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6238556504249573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7269612550735474 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18413574993610382 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03841531276702881 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6625620722770691 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.06353089958429337 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7129203081130981 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.214729443192482 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚6 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20964796841144562 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4073270261287689 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_3_of_3‚0.9995090961456299 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7663313746452332 /particle/accuracy_4_of_4‚0.9448955059051514 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10072232782840729 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7998597025871277 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7532916069030762 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969BlÏ;Ý=± ×{ (particle/accuracy‚0.8421921730041504 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3817979693412781 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04202000051736832 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6453130841255188 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17417219281196594 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2766585946083069 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5803357362747192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1877547651529312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 'particle/f_score‚0.6206983327865601 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4912283718585968 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0430738590657711 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6418932676315308 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 +jet/accuracy_4_of_4‚0.01628045178949833 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7673410773277283 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6422370076179504 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2244926393032074 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18671837449073792 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1304996758699417 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213908672332764 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18321841955184937 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8084079027175903 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9878384470939636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21162793040275574 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4191724359989166 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3083656430244446 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_3_of_5‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02344956435263157 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09844423830509186 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937272071838379 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1662534475326538 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1733028143644333 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962902069091797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 )loss/val_total_loss‚8.980026540261347 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3759167790412903 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.665765106678009 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.038429636508226395 *jet/accuracy_2_of_3‚0.5085383653640747 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20350582897663116 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953615069389343 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6443200707435608 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16890081763267517 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18662221729755402 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.799622118473053 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 *jet/accuracy_2_of_4‚0.7349887490272522 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955456256866455 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7991392016410828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8503969311714172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6207932829856873 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.829176127910614 .particle/accuracy_3_of_4‚0.997619092464447 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8203892707824707 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012263154610991478 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.038429636508226395 *particle/specificity‚0.908067524433136 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18737079203128815 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5917336344718933 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚2799 _runtime‚2399.168275124 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11725087463855743 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6238556504249573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7269612550735474 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18413574993610382 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03841531276702881 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6625620722770691 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.06353089958429337 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7129203081130981 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.214729443192482 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚6 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20964796841144562 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4073270261287689 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_3_of_3‚0.9995090961456299 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7663313746452332 /particle/accuracy_4_of_4‚0.9448955059051514 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10072232782840729 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7998597025871277 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7532916069030762 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04457166790962219 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6421090364456177 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6452142596244812 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6926180124282837 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7016486525535583 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04262131080031395 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942910075187683 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6607264280319214 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8712705969810486 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9895304441452026 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6443200707435608 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08199667930603027 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.993288516998291 *jet/accuracy_1_of_4‚0.9792044758796692 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.837154746055603 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2232438027858734 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9326568841934204 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6273972988128662 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 particle/accuracy_2_of_2‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19749654829502106 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6024418473243713 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 *jet/accuracy_1_of_2‚0.7761469483375549 jet/accuracy_1_of_5‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49559611082077026 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20778630673885345 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6512986421585083 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7070336937904358 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7471150755882263 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6901841163635254 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4385921061038971 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/sensitivity‚0.6013011336326599 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2707775831222534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2193213701248169 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7375200390815735 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 *jet/accuracy_1_of_3‚0.9122800230979919 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.27658164501190186 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6957243084907532 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922221899032593 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6925548911094666 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6555728912353516 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7496275901794434 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8388512134552002 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946234822273254 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3513830602169037 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7553712129592896 +jet/accuracy_1_of_1‚0.32302480936050415 7validation_average_jet_accuracy‚0.36203211545944214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.05483259633183479 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7318395972251892 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020174413919448853 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18654386699199677 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6582232713699341 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7304043769836426 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚230 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934182167053223 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3269789218902588 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04563792794942856 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.837154746055603 % _timestamp‚1.7843427404739008e+09 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039606425911188126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16234450042247772 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21162793040275574 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1æ~·‘² ‚(:’ ÕÑëÒ½«š“ gpu.0.gpu‚90.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.287' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.05739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M9tÝó ‚(:¸ ÕÑëÒ²¤Š• disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚7129.66357421875 disk.rbd1.in‚0 network.sent‚ 4.313335e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚7129.66357421875* disk./.usagePercent‚39.888992419473546 disk.dm-0.in‚ 5536.09765625 disk.sda2.out‚0 disk.rbd1.out‚ 6.01953125 disk.dm-1.in‚0 disk.dm-1.out‚ 62.3828125! disk.nvme0n1.in‚ 5536.09765625& proc.memory.availableMB‚ 489.796875# disk./.usageGB‚594.374813079834 disk.rbd0.in‚0 disk.sda.out‚ 62.3828125% disk.nvme0n1.out‚7129.66357421875 disk.sda3.in‚0# proc.memory.rssMB‚ 2724.30859375 network.recv‚666004 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 62.3828125$ memory_percent‚97.01051712036133 cpu‚26.17342025192746# disk.nvme0c0n1.in‚ 5536.09765625* proc.memory.percent‚16.627860069274902 proc.cpu.threads‚17¤“´ ‚(:ˆ äÑëÒéƒÝ” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚206.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï“žµ ‚(:³ äÑëÒ™•æ• disk.rbd1.in‚0 disk.rbd1.out‚ 6.01953125 disk.dm-1.in‚0 cpu‚26.170805194025967$ disk./.usageGB‚594.3748893737793 disk.rbd0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 489.89453125 network.sent‚ 4.369724e+06& disk.nvme0c0n1.out‚7133.9736328125! disk.dm-0.out‚7133.9736328125 disk.sda.out‚62.65625 disk.sda2.out‚0$ disk.nvme0n1.out‚7133.9736328125# proc.memory.rssMB‚ 2724.30859375 disk.drbd2165.out‚0 network.recv‚669318* disk./.usagePercent‚39.888997539624235! disk.nvme0n1.in‚ 5677.55078125 disk.sda3.out‚62.65625 disk.rbd0.out‚0 disk.dm-0.in‚ 5677.55078125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0$ memory_percent‚97.00992107391357* proc.memory.percent‚16.627860069274902# disk.nvme0c0n1.in‚ 5677.55078125 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚62.65625 disk.drbd2165.in‚0­¸‰›“¶ ‚(:ˆ óÑëÒ󲚔 gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t’ÙžZ· ‚(:º óÑëҳ؇—* proc.memory.percent‚16.627860069274902 proc.cpu.threads‚ A°k17) disk./.usagePercent‚39.88899011540574# memory_percent‚97.0099687576294# proc.memory.rssMB‚ 2724.30859375 cpu‚26.167905228180185$ disk./.usageGB‚594.3747787475586 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 62.83984375 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 5677.9921875 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 5677.9921875" disk.dm-0.out‚7145.12744140625 network.sent‚ 4.374658e+06 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7145.12744140625 disk.sda2.out‚0% disk.nvme0n1.out‚7145.12744140625 disk.sda3.out‚ 62.83984375( proc.memory.availableMB‚ 489.88671875 network.recv‚672708 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 62.83984375 disk.nvme0n1.in‚ 5677.9921875 disk.rbd1.out‚ 6.14453125ƒŠŽ2¸ - _wandbruntime‚2438  _runtime‚2438ô³Ïɹ à ' lr-AdamW/pg1‚0.00043199873944902285 ' lr-AdamW/pg2‚0.00043199873944902285 trainer/global_step‚2849 % _timestamp‚1.7843427799877825e+09 _runtime‚2438.681342357 _step‚231 a±þκ È % _timestamp‚1.7843427799877825e+09 _runtime‚2438.681342357 _step‚231 ' lr-AdamW/pg1‚0.00043199873944902285 ' lr-AdamW/pg2‚0.00043199873944902285 trainer/global_step‚2849ç÷gÄ2» - _wandbruntime‚2438  _runtime‚2438™y·Ü'¼ ÖO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708368896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6132781 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450355712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6133042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433179741986816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 % _timestamp‚1.7843427799888506e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436194450355712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98555015151104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091977216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430951341150208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708368896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091977216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599724 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98555120243200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _runtime‚2438.682728023 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26392823 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392823 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3014367029760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95507325380608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367029760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129813 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4996768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171069¶b“cá'½ ÛO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _runtime‚2438.682728023 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26392823 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392823 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3014367029760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95507325380608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367029760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129813 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4996768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171069 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708368896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6132781 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450355712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6133042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433179741986816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 % _timestamp‚1.7843427799888506e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436194450355712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98555015151104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091977216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430951341150208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708368896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091977216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599724 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98555120243200è»õØ 2¾ - _wandbruntime‚2439  _runtime‚2439 "ìÙæ&¿ àM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009229 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2849 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436342704019456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401875 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843427806039474e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589469798400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26411866 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3011120074695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434108971896775 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379656704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22607354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6135148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434108614307413 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3015379656704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95540754139136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11133664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524693 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.al„Õûl#located‚11133664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524693 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401875 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843427806039474e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589469798400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26411866 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3011120074695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434108971896775 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379656704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22607354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6135148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269é¯@ˆÏ2Á - _wandbruntime‚2439  _runtime‚2439 Þ € epoch‚7 0loss/FBt2/detection_loss‚0.45155780463803397 0loss/FRt1/assignment_loss‚1.8885051693787447 0loss/SRqqt2/detection_loss‚0.613116044331736 % _timestamp‚1.7843427806065416e+09 _step‚234 /loss/FRt2/assignment_loss‚1.015662142952669 1loss/SRqqt1/assignment_loss‚0.616974413279024 2loss/SRqqt2/assignment_loss‚0.5390624986667207 2loss/FBt1/assignment_loss‚0.022595968582929043 /loss/FRt1/detection_loss‚0.5980270193060396 /loss/FRt2/detection_loss‚1.2708005853964794 /loss/FBt1/detection_loss‚1.1046703909135733 1loss/FBt2/assignment_loss‚0.03699222762608831 1loss/SRqqt1/detection_loss‚0.9636602194640583 %loss/total_loss‚9.121624484536095 trainer/global_step‚2849 _runtime‚2439.299823461·»à à … epoch‚7 0loss/FBt2/detection_loss‚0.45155780463803397 0loss/FRt1/assignment_loss‚1.8885051693787447 0loss/SRqqt2/detection_loss‚0.613116044331736 % _timestamp‚1.7843427806065416e+09 _step‚234 /loss/FRt2/assignment_loss‚1.015662142952669 1loss/SRqqt1/assignment_loss‚0.616974413279024 2loss/SRqqt2/assignment_loss‚0.5390624986667207 2loss/FBt1/assignment_loss‚0.022595968582929043 /loss/FRt1/detection_loss‚0.5980270193060396 /loss/FRt2/detection_loss‚1.2708005853964794 /loss/FBt1/detection_loss‚1.1046703909135733 1loss/FBt2/assignment_loss‚0.03699222762608831 1loss/SRqqt1/detection_loss‚0.9636602194640583 %loss/total_loss‚9.121624484536095 trainer/global_step‚2849 _runtime‚2439.299823461êóù?žÄ ‚(:“ ‚ÒëÒÆ´«” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€cßÆÅ ‚(:» ‚ÒëÒŠ•¦–! disk.nvme0n1.in‚ 5678.82421875# disk.nvme0c0n1.in‚ 5678.82421875 disk.dm-0.in‚ 5678.82421875# proc.memory.rssMB‚ 2724.30859375* proc.memory.percent‚16.627860069274902 cpu‚26.16518543052273 network.recv‚675970 disk.sda.out‚ 64.55859375 disk.rbd1.out‚ 6.14453125 disk.rbd0.in‚0" disk.dm-0.out‚7157.72900390625 disk.sda.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚7157.72900390625$ memory_percent‚97.01180458068848 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7157.72900390625' proc.memory.availableMB‚ 489.5859375( disk./.usagePercent‚39.8887983657626 disk.sda3.out‚ 64.55859375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 64.55859375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 network.sent‚ 4.379629e+06$ disk./.usageGB‚594.3719215393066 disk.sda3.in‚0 disk.rbd1.in‚0×*Æ ‚(:’ ‘ÒëÒ؀ٓ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Ó‘ßÊÇ ‚(:¿ ‘ÒëÒÙÞî•( proc.memory.availableMB‚ 489.66015625 network.recv‚680524 disk.rbd1.out‚ 6.20703125 disk.dm-0.in‚ 5678.82421875 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 5678.82421875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 5678.82421875$ memory_percent‚97.01135158538818 disk.sda1.in‚0* disk./.usagePercent‚39.888814750244784$ disk./.usageGB‚594.3721656799316% disk.nvme0n1.out‚7163.92138671875" disk.dm-0.out‚7163.92138671875 network.sent‚ 4.442587e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7163.92138671875 disk.sda.out‚ 65.11328125 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 65.11328125 disk.sda.in‚0# proc.memory.rssMB‚ 2724.30859375 cpu‚26.162391318560246 disk.rbd0.in‚0 disk.dm-1.out‚ 65.11328125* proc.memory.percent‚16.627860069274902J±WLÈ ‚(:’  ÒëҴϬ“ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¼›’ÄÉ ‚(:¹  ÒëÒﺔ– disk.dm-1.out‚ 65.76171875! disk.dm-0.out‚7171.6240234375$ disk.nvme0n1.out‚7171.6240234375 disk.rbd0.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 5692.04296875 disk.sda3.in‚0 disk.rbd1.out‚ 6.20703125 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 65.76171875 disk.sda1.in‚0 disk.dm-0.in‚ 5692.04296875# proc.memory.rssMB‚ 2724.30859375 network.sent‚ 4.447424e+06# disk./.usageGB‚594.371711730957 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚97.01132774353027 network.recv‚683878) disk./.usagePercent‚39.88878428534821 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 5692.04296875 disk.sda.in‚0 disk.sda3.out‚ 65.76171875 disk.rbd1.in‚0* proc.memory.percent‚16.627860069274902 cpu‚26.159814653736884& disk.nvme0c0n1.out‚7171.6240234375' proc.memory.availableMB‚ 489.6640625 proc.cpu.threads‚17ª@2Ê - _wandbruntime‚2478  _runtime‚24787‡…ÈË Â _runtime‚2478.551697574 _step‚235 ' lr-AdamW/pg1‚0.00043130370776985305 ' lr-AdamW/pg2‚0.00043130370776985305 trainer/global_step‚2899 $ _timestamp‚1.784342819857977e+09§š¢çÍÌ Ç ' lr-AdamW/pg1‚0.00043130370776985305 ' lr-AdamW/pg2‚0.00043130370776985305 trainer/global_step‚2899 $ _timestamp‚1.784342819857977e+09 _runtime‚2478.551697574 _step‚235ëQQiZ2Í - _wandbruntime‚2478  _runtime‚2478ìª<Ý'Î ×O VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403870720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440541793468416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22981322 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440541776429056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847275 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838960640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062531584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6237221 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403870720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11321497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6237476 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857252 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277990417408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976DoÙšßÏ ÜO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26847275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251577 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚236 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277990417408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403870720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440541793468416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22981322 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440541776429056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847275 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838960640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segm$P‹4 ents.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062531584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6237221 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403870720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11321497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6237476 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857252 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11325094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843428204755077e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856327 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26856327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440689034498048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3061736561603 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100259735040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443755451002880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416504832 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6239331 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441484764387921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440689017458688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866318 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6239581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3100179553792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011601 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451002880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚2479.169585519 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440689017458688Å3=ì&Ò æM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451002880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚2479.169585519 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440689017458688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11325347 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚237 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440689034498048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22988954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11325094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843428204755077e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856327 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26856327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440689034498048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3061736561603 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100259735040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443755451002880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416504832 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6239331 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441484764387921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440689017458688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866318 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6239581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3100179553792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011601 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664649 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6341377 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27301727 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 _runtime‚2518.690416699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099372544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3150428339200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758033408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3111001444787 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150512721920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585944064 SGDeviceStatsMonitor.on_train_batch_start/inactive_–ÚëCsplit.small_pool.freed‚6341108 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332085 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944322560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11512376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000713955328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27311704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758033408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000610960384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742835 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27301727 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5171268 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‘WÜã'ß ÝO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27301727 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5171268 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664649 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6341377 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27301727 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 _runtime‚2518.690416699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099372544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3150428339200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758033408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3111001444787 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150512721920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585944064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6341108 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332085 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944322560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11512376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000713955328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27311704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758033408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000610960384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742835ð¼Æj2à - _wandbruntime‚2519  _runtime‚2519 =ÐÕç&á áM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23370556 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3112012498509 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770660352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6343446 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23370554 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112006656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3151443607040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27310779 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057791 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048511 % _timestamp‚1.7843428606095905e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691326 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451168197986304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839600640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11516211 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112006656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 2519.30339699 _step‚241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11516462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23370556 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310779 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102035164265984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691326 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448051085979648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0/Ó×Éâ æM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112006656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3151443607040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27310779 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057791 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048511 % _timestamp‚1.7843428606095905e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691326 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451168197986304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839600640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11516211 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112006656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 2519.30339699 _step‚241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11516462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23370556 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310779 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102035164265984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691326 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448051085979648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23370556 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3112012498509 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770660352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6343446 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23370554 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ñ{Ë_ 2ã - _wandbruntime‚2519  _runtime‚2519ð×êCä ‚ /loss/FBt1/detection_loss‚0.7164148317856921 0loss/FBt2/detection_loss‚0.49495235440469737 trainer/global_step‚2949 _runtime‚2519.304909106 0loss/FRt2/assignment_loss‚0.9310168699142354 1loss/FBt2/assignment_loss‚0.07691767656048444 /loss/FRt2/detection_loss‚1.1956541955915776 %loss/total_loss‚8.759509071558325 _step‚242 0loss/FRt1/assignment_loss‚1.9286286639516776 epoch‚7 2loss/SRqqt2/assignment_loss‚0.6054728310801865 /loss/FRt1/detection_loss‚0.5860430801077169 % _timestamp‚1.7843428606116378e+09 2loss/SRqqt1/assignment_loss‚0.6751843614861773 1loss/FBt1/assignment_loss‚0.05211458983623787 1loss/SRqqt1/detection_loss‚0.8497558305297727 1loss/SRqqt2/detection_loss‚0.6473537863098684j±‡ å ‡ /loss/FRt2/detection_loss‚1.1956541955915776 %loss/total_loss‚8.759509071558325 _step‚242 0loss/FRt1/assignment_loss‚1.9286286639516776 epoch‚7 2loss/SRqqt2/assignment_loss‚0.6054728310801865 /loss/FRt1/detection_loss‚0.5860430801077169 % _timestamp‚1.7843428606116378e+09 2loss/SRqqt1/assignment_loss‚0.6751843614861773 1loss/FBt1/assignment_loss‚0.05211458983623787 1loss/SRqqt1/detection_loss‚0.8497558305297727 1loss/SRqqt2/detection_loss‚0.6473537863098684 /loss/FBt1/detection_loss‚0.7164148317856921 0loss/FBt2/detection_loss‚0.49495235440469737 trainer/global_step‚2949 _runtime‚2519.304909106 0loss/FRt2/assignment_loss‚0.9310168699142354 1loss/FBt2/assignment_loss‚0.07691767656048444òˆÍ»žæ ‚(:“ ÍÒëÒÖ¶·“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.372600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â+ ÝÆç ‚(:» ÍÒëÒÔÙŠ— disk.rbd0.out‚0 disk.dm-1.out‚ 67.1640625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 5781.19140625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7194.81884765625$ memory_percent‚97.01383113861084( proc.memory.availableMB‚ 489.25390625* proc.memory.percent‚16.627860069274902 proc.cpu.threads‚17 network.sent‚ 4.520292e+06 network.recv‚695052 disk.sda.out‚ 67.1640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2724.30859375" disk.dm-0.out‚7194.81884765625 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 cpu‚26.15177435937871$ disk./.usageGB‚594.3710746765137 disk.dm-0.in‚ 5781.19140625 disk.sda3.out‚ 67.1640625 disk.rbd1.out‚ 6.33203125 disk.sda1.in‚0! disk.nvme0n1.in‚ 5781.19140625% disk.nvme0n1.out‚7194.81884765625 disk.sda3.in‚0* disk./.usagePercent‚39.888741532089995zðœè ‚(:… ÜÒëÒÒ›” gpu.0.gpu‚17.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nCàÅé ‚(:º ÜÒëÒïÚÊ–) disk./.usagePercent‚39.88874358015026% disk.nvme0n1.out‚7202.91064453125$ memory_percent‚97.01380729675293# proc.memory.rssMB‚ 2724.30859375* proc.memory.percent‚16.627860069274902 disk.rbd1.out‚ 6.33203125' proc.memory.availableMB‚ 489.2578125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 5782.0234375 disk.nvme0n1.in‚ 5782.0234375$ disk./.usageGB‚594.3711051940918 disk.rbd0.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚7202.91064453125 disk.sda3.in‚0 disk.sda3.out‚ 67.43359375 network.sent‚ 4.583175e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 67.43359375 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 67.43359375 disk.sda2.in‚0 disk.rbd1.in‚0 network.recv‚699119 disk.drbd2165.out‚0 disk.sda2.out‚0 cpu‚26.149337864784723 disk.dm-1.in‚0 disk.dm-0.in‚ 5782.0234375" disk.dm-0.out‚7202.91064453125 proc.cpu.threads‚17ޘȓê ‚(:ˆ ëÒëÒÃÿ³“ gpu.0.gpu‚64.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì©I”Åë ‚(:º ëÒëÒřǖ network.sent‚ 4.588123e+06 disk.sda3.in‚0 disk.rbd1.out‚ 6.39453125 disk.rbd0.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5884.96484375! disk.nvme0n1.in‚ 5884.96484375# proc.memory.rssMB‚ 2724.30859375* proc.memory.percent‚16.627860069274902 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 67.7109375 cpu‚26.14691781372932 proc.cpu.threads‚17$ disk./.usageGB‚594.3713722229004 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 67.7109375' disk.nvme0c0n1.out‚7209.07080078125$ memory_percent‚97.01297283172607( proc.memory.availableMB‚ 489.39453125 network.recv‚702468) disk./.usagePercent‚39.88876150067766 disk.dm-0.in‚ 5884.96484375 disk.sda.out‚ 67.7109375% disk.nvme0n1.out‚7209.07080078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚7209.07080078125 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.in‚0ñ­'ÿ2ì - _wandbruntime‚2558  _runtime‚2558<9êÉí à ' lr-AdamW/pg1‚0.00042987384947076553 ' lr-AdamW/pg2‚0.00042987384947076553 trainer/global_step‚2999 % _timestamp‚1.7843429000203586e+09 _runtime‚2558.713932497 _step‚2434ñŽÎî È _step‚243 ' lr-AdamW/pg1‚0.00042987384947076553 ' lr-AdamW/pg2‚0.00042987384947076553 trainer/global_step‚2999 % _timestamp‚1.7843429000203586e+09 _runtime‚2558.713932497ó…Ú¾µ2ï - _wandbruntime‚2558  _runtime‚2558¶–=Và'ð ÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500701 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23744522 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455265879392256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6445151 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6444881 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚244 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166453535232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794874368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23744524 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 trainer/global_step‚2999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11703399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931695 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691305984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458432332927488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5258521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103723202187264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691305984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766156 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455265896431616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756179 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201666022400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166453535232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756179 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332927488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481585 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚2558.715364711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794874368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3201581639680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452927924891648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27766156 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103723305182208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103723305182208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103723202187264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691305984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766156 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455265896431616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756179 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201666022400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166453535232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756179 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332927488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481585 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚2558.715364711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794874368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136ôªù-A2ò - _wandbruntime‚2559  _runtime‚2559x>¾i ó ãM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527378 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535419 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775222 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23752156 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11707454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _step‚245 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11707194 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27765231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23752154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3167807508480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466162176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944969728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784342900635903e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453074435563520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202607927808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586584064 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6446931 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944969728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚2999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27765231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757760221696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0sW»eî&ô èM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985417 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27775222 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3167807508480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527378 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517385 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757657233920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517385 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2559.329687513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775222 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23752156 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11707454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _step‚245 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11707194 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27765231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23752154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3167807508480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466162176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944969728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784342900635903e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453074435563520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202607927808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586584064 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6446931 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944969728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚2999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27765231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757760221696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527378 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535419 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138299 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079910912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6548199 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336753 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28220608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462627947913216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438289408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463465692903851 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105445927469056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252766000128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212234418603 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968493 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24126122 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28220608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465845438289408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465845079910912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24126124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDevicˆãáãMeStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079910912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6548199 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336753 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28220608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462627947913216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438289408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463465692903851 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105445927469056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252766000128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212234418603 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968493 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24126122 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28220608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465845438289408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465845079910912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24126124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚248 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _step‚249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462775171903488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333567488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465993691953152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3218503010304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462775171903488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161664000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784342940512719e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6550551 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105480267456000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2599.206516867 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219683 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3218503010304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102226487232512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5347513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253870890496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3253780223488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353437 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11897807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218807 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360°í0uî&† èM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784342940512719e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6550551 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105480267456000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2599.206516867 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219683 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3218503010304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102226487232512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5347513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253870890496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3253780223488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353437 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11897807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218807 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24133756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462775188942848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6550294 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _step‚249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462775171903488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333567488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465993691953152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3218503010304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462775171903488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161664000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826894336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168591364608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185272832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12084900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473257826894336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24507724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303998260736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905511 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185272832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843429799037256e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844538880 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651882 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24507722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469989982355456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470842200573609 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103864515012608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3304076352000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12085160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172805 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107168494661120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6652139 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984359 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208 _runtime‚2638.597496702 _step‚252 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675060 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844538880 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455401 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469989982355456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185878016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257ZLgä'“ ÞO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6652139 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984359 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208 _runtime‚2638.597496702 _step‚252 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675060 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844538880 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455401 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469989982355456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185878016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826894336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168591364608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185272832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12084900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473257826894336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24507724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303998260736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905511 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185272832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843429799037256e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844538880 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651882 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24507722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469989982355456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470842200573609 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103864515012608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3304076352000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12085160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172805 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107168494661120üK•—2” - _wandbruntime‚2639  _runtime‚2639_ÛŸS$• âM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189489 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12088733 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674135 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24515356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470137240424448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189489 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12088988 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509235 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784342980518043e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2639.211849599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473406438936576 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28674135 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6654222 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438936576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107202956509184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3305107337216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3099 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198512128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3305031350272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709953 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24515354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3268857165824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107203051108352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24515354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28684126 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198Ï‹Ê•512128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24515356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857165824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299315É‚ìí&– çM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198512128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3305031350272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709953 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24515354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3268857165824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107203051108352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24515354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28684126 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198512128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24515356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857165824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299315 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189489 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12088733 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674135 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24515356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470137240424448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189489 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12088988 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509235 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784342980518043e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2639.211849599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473406438936576 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28674135 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6654222 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438936576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107202956509184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3305107337216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3099 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843430199711916e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3318881379840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018836 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29129512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480670932256256 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3355389869568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932256256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108891342808576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881379840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3313467392419 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573877760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚477352033837056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24889322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 _step‚256 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3318540040704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24889324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573877760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0qEê¥ ßO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932256256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108891342808576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881379840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3313467392419 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573877760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4773Û"dû52033837056 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24889322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 _step‚256 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3318540040704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24889324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573877760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843430199711916e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3318881379840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018836 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29129512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480670932256256 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3355389869568€UtpV2¦ - _wandbruntime‚2679  _runtime‚2679p R#ê&§ äM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035534 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6757272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128587 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894013952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3319552667648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6757003 trainer/global_step‚3149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379823 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790461 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3319894013952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12279016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025541 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475051036344320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477499291906048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477499274866688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5522013 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827534336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12279288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925689664512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3356326579200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925792652288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3356410954752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843430205854647e+09 _runtime‚2679.279297413 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54025541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12279016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025541 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475051036344320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477499291906048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477499274866688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5522013 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827534336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12279288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925689664512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3356326579200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925792652288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3356410954752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843430205854647e+09 _runtime‚2679.279297413 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54025541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6757003 trainer/global_step‚3149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379823 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790461 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3319894013952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679239680 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3369235542528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2718.459376468 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844909 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843430597619548e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320861184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485594143145375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3369576881664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12466024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484714085318656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3406601122816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3406504157184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3369235542528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5607521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858506 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734625 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484714102358016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6858781 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269—XÞuå'· ßO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5607521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858506 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734625 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484714102358016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6858781 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488083679239680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3364424429217 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488083320861184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12466302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679239680 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3369235542528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2718.459376468 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844909 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843430597619548e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320861184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485594143145375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3369576881664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12466024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484714085318656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3406601122816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3406504157184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3369235542528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518„i‡—L2¸ - _wandbruntime‚2719  _runtime‚2719m¹8ùê&¹ äM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484861326348288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861593 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚3199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370589515776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583039 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460331 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484861326348288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248169472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483051 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3407523156480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5609266 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12470082 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589515776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25278556 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3365094933049 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29593030 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932903424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110648324168192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861593 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991Nhd0º éM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932903424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110648324168192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861593 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484861326348288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861593 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚3199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370589515776 NADeviceStatsMonitor.on_train_batch_end/allocati±r¿on.small_pool.freed‚29583039 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460331 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484861326348288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248169472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870969 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483051 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3407523156480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5609266 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12470082 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589515776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25278556 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3365094933049 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29593030 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0…ð2» - _wandbruntime‚2719  _runtime‚2719…0Ú¼ ÿ 2loss/SRqqt2/assignment_loss‚0.5145108460756882 1loss/FBt1/assignment_loss‚0.10877153548275623 /loss/FRt2/detection_loss‚1.2716183433494734 0loss/FRt1/assignment_loss‚1.8209632039725125 1loss/SRqqt1/detection_loss‚1.0447877158362675 /loss/FBt1/detection_loss‚1.1433256105359257 /loss/FBt2/detection_loss‚0.6737561507104295 %loss/total_loss‚9.500808149201497 /loss/FRt2/assignment_loss‚0.984345460892655 1loss/SRqqt1/assignment_loss‚0.663123016970198 2loss/FBt2/assignment_loss‚0.010070918221511632 /loss/FRt1/detection_loss‚0.6358874628882983 1loss/SRqqt2/detection_loss‚0.6296478842657803 epoch‚7 trainer/global_step‚3199 _runtime‚2719.071869881 $ _timestamp‚1.784343060378561e+09 _step‚262ª¬2Ó ½ „ 0loss/FRt1/assignment_loss‚1.8209632039725125 1loss/SRqqt1/detection_loss‚1.0447877158362675 /loss/FBt1/detection_loss‚1.1433256105359257 /loss/FBt2/detection_loss‚0.6737561507104295 %loss/total_loss‚9.500808149201497 /loss/FRt2/assignment_loss‚0.984345460892655 1loss/SRqqt1/assignment_loss‚0.663123016970198 2loss/FBt2/assignment_loss‚0.010070918221511632 /loss/FRt1/detection_loss‚0.6358874628882983 1loss/SRqqt2/detection_loss‚0.6296478842657803 epoch‚7 trainer/global_step‚3199 _runtime‚2719.071869881 $ _timestamp‚1.784343060378561e+09 _step‚262 2loss/SRqqt2/assignment_loss‚0.5145108460756882 1loss/FBt1/assignment_loss‚0.10877153548275623 /loss/FRt2/detection_loss‚1.2716183433494734†¬.'’¾ ‚(:‡ ŸÔëÒú±®” gpu.0.gpu‚20.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚286.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽçÄ·µ¿ ‚(:ª ŸÔëÒÀΕ* proc.memory.percent‚16.603684425354004 cpu‚26.13363480441566 disk.drbd2165.out‚0! disk.dm-0.out‚8207.2646484375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚73.5625) disk./.usagePercent‚39.89223526690977& disk.nvme0c0n1.out‚8207.2646484375 disk.dm-0.in‚ 6230.10546875 disk.rbd1.out‚ 6.95703125# proc.memory.rssMB‚ 2720.34765625$ disk.nvme0n1.out‚8207.2646484375 disk.rbd1.in‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚491.3125 network.recv‚747579$ disk./.usageGB‚594.4231338500977 disk.sda3.in‚0 disk.sda3.out‚73.5625 disk.dm-1.in‚0 network.sent‚ 4.879637e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6230.10546875! disk.nvme0n1.in‚ 6230.10546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚73.5625$ memory_percent‚97.00126647949219Éýª2À - _wandbruntime‚2736  _runtime‚2736 o”•=Á { 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22407856583595276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872878193855286 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2593792974948883 epoch‚7 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6637465953826904 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9900762438774109 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _step‚263 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695747375488281 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9730769395828247 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9129670262336731 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25694185495376587 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024092700332403183 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8044806718826294 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23593156039714813 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7677642107009888 ) Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05633927136659622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21566686034202576 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.772331953048706 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6587164998054504 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04508864879608154 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7859591245651245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.970140278339386 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8226252198219299 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45370373129844666 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918883442878723 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5913885235786438 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859391450881958 *jet/accuracy_1_of_2‚0.7888107299804688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036043792963027954 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02719590812921524 (particle/accuracy‚0.8356545567512512 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029846489429473877 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6328657269477844 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8423659205436707 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6695781350135803 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +particle/specificity‚0.9020811915397644 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8532797694206238 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7686589360237122 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49306386709213257 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5680760741233826 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980305433273315 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25438645482063293 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20605868101119995 +jet/accuracy_3_of_3‚0.04045041278004646 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11587220430374146 4 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914590716362 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042311184108257294 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0414857491850853 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7397006154060364 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6347172856330872 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7619085907936096 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.799357533454895 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907785058021545 *jet/accuracy_1_of_4‚0.9730935096740723 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6660457253456116 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6597123742103577 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37877705693244934 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7269653081893921 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6980319619178772 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8652873039245605 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032942093908786774 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02719590812921524 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8984021544456482 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6642126441001892 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7158058285713196 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9815695881843567 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9835034608840942 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25694185495376587 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6076796650886536 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074071645736694 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ) Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7639038562774658 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 trainer/global_step‚3199 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9536218047142029 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4788799285888672 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26678916811943054 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39126986265182495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21863548457622528 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7461069822311401 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7354947328567505 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6587778329849243 /particle/accuracy_3_of_4‚0.9880669116973877 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09794456511735916 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7662549018859863 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8629429340362549 *jet/accuracy_1_of_3‚0.9261265993118286 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3773101270198822 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33063212037086487 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4711601734161377 particle/accuracy_2_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20391476154327393 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6511180400848389 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7060112357139587 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5800579786300659 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06785683333873749 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24298542737960815 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7860055565834045 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9375443458557129 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24783307313919067 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.863000750541687 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032700810581445694 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16211850941181183 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8520786762237549 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333432674408 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21984754502773285 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21634002029895782 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09792239218950272 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.707386314868927 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20251762866973877 *jet/accuracy_2_of_3‚0.4947364926338196 *jet/accuracy_2_of_4‚0.7493391633033752 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30575165152549744 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6884049773216248 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 % _timestamp‚1.7843430777593327e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20784573256969452 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15421047806739807 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7131454944610596 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6420745253562927 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6597123742103577 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2688668966293335 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9932734966278076 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19692552089691162 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958730340003967 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7679488658905029 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22535169124603271 )loss/val_total_loss‚8.972317653841289 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21684005856513977 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074071645736694 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30662399530410767 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030959513038396835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9897533655166626 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7692734003067017 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8629429340362549 +jet/accuracy_3_of_4‚0.04310102015733719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9768108129501343 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8214870691299438 +jet/accuracy_2_of_2‚0.12448213249444962 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997155487537384 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6959609985351562 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0801592767238617 +jet/accuracy_1_of_1‚0.34373560547828674 *particle/sensitivity‚0.592744767665863 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2736.454052893 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1954973340034485 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47222647070884705 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18675753474235535,³ ”{ +particle/specificity‚0.9020811915397644 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8532797694206238 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7686589360237122 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49306386709213257 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5680760741233826 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980305433273315 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25438645482063293 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20605868101119995 +jet/accuracy_3_of_3‚0.04045041278004646 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11587220430374146 4 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9914590716362 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042311184108257294 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0414857491850853 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7397006154060364 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6347172856330872 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7619085907936096 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.799357533454895 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907785058021545 *jet/accuracy_1_of_4‚0.9730935096740723 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6660457253456116 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6597123742103577 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37877705693244934 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7269653081893921 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6980319619178772 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8652873039245605 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032942093908786774 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02719590812921524 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8984021544456482 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6642126441001892 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7158058285713196 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9815695881843567 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9835034608840942 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25694185495376587 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6076796650886536 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074071645736694 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ) Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7639038562774658 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 trainer/global_step‚3199 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9536218047142029 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4788799285888672 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26678916811943054 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39126986265182495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21863548457622528 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7461069822311401 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7354947328567505 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6587778329849243 /particle/accuracy_3_of_4‚0.9880669116973877 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09794456511735916 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7662549018859863 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8629429340362549 *jet/accuracy_1_of_3‚0.9261265993118286 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3773101270198822 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33063212037086487 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4711601734161377 particle/accuracy_2_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20391476154327393 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6511180400848389 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7060112357139587 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5800579786300659 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06785683333873749 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24298542737960815 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7860055565834045 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9375443458557129 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24783307313919067 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.863000750541687 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032700810581445694 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16211850941181183 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8520786762237549 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333432674408 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21984754502773285 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21634002029895782 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09792239218950272 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.707386314868927 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20251762866973877 *jet/accuracy_2_of_3‚0.4947364926338196 *jet/accuracy_2_of_4‚0.7493391633033752 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBtÿžTç/event_purity‚0.30575165152549744 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6884049773216248 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 % _timestamp‚1.7843430777593327e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20784573256969452 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15421047806739807 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7131454944610596 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6420745253562927 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6597123742103577 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2688668966293335 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9932734966278076 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19692552089691162 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958730340003967 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7679488658905029 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22535169124603271 )loss/val_total_loss‚8.972317653841289 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21684005856513977 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074071645736694 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30662399530410767 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030959513038396835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9897533655166626 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7692734003067017 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8629429340362549 +jet/accuracy_3_of_4‚0.04310102015733719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9768108129501343 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8214870691299438 +jet/accuracy_2_of_2‚0.12448213249444962 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997155487537384 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6959609985351562 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0801592767238617 +jet/accuracy_1_of_1‚0.34373560547828674 *particle/sensitivity‚0.592744767665863 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2736.454052893 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1954973340034485 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47222647070884705 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18675753474235535 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22407856583595276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872878193855286 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2593792974948883 epoch‚7 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6637465953826904 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9900762438774109 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _step‚263 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695747375488281 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9730769395828247 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9129670262336731 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25694185495376587 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024092700332403183 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8044806718826294 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23593156039714813 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7677642107009888 ) Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05633927136659622 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21566686034202576 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.772331953048706 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6587164998054504 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04508864879608154 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7859591245651245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.970140278339386 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8226252198219299 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45370373129844666 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918883442878723 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5913885235786438 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859391450881958 *jet/accuracy_1_of_2‚0.7888107299804688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036043792963027954 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02719590812921524 (particle/accuracy‚0.8356545567512512 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029846489429473877 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6328657269477844 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8423659205436707 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6695781350135803 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075‡Žq¡“à ‚(:ˆ ®ÔëÒïÿ“ gpu.0.gpu‚77.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{j (½Ä ‚(:² ®ÔëÒ‰ýÞ• disk.dm-1.in‚0$ memory_percent‚95.47736644744873 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 8461.33984375) disk./.usagePercent‚39.89231206917004 disk.sda1.in‚0 disk.sda.out‚ 73.80859375 disk.nvme0n1.in‚ 6231.0703125$ disk./.usageGB‚594.4242782592773 disk.sda1.out‚0 disk.dm-1.out‚ 73.80859375" disk.nvme0c0n1.in‚ 6231.0703125 disk.sda.in‚0* proc.memory.percent‚16.627860069274902 network.sent‚ 4.942495e+06 disk.sda3.out‚ 73.80859375 disk.rbd1.out‚ 6.95703125" disk.nvme0n1.out‚ 8461.33984375# proc.memory.rssMB‚ 2724.30859375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 8461.33984375( proc.memory.availableMB‚ 740.98828125 cpu‚26.139038459674513 disk.dm-0.in‚ 6231.0703125 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚752002 disk.rbd0.in‚0? &žÅ ‚(:“ ½ÔëÒí§“ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚‡¶Ž¿Æ ‚(:´ ½ÔëÒÙá• disk.nvme0n1.in‚ 6231.0703125 disk.sda3.in‚0 network.sent‚ 4.999598e+06 network.recv‚755784 disk.sda.out‚ 74.171875% disk.nvme0n1.out‚8504.57958984375( proc.memory.availableMB‚ 740.92578125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 6231.0703125 disk.dm-0.in‚ 6231.0703125 disk.sda3.out‚ 74.171875# proc.memory.rssMB‚ 2724.30859375 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.08203125* proc.memory.percent‚16.627860069274902 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚8504.57958984375 disk.rbd1.in‚0# memory_percent‚95.4777479171753 disk.dm-1.out‚ 74.171875' disk.nvme0c0n1.out‚8504.57958984375 cpu‚26.136766357801328 proc.cpu.threads‚17) disk./.usagePercent‚39.89223526690977$ disk./.usageGB‚594.4231338500977 disk.sda.in‚0½¶yážÇ ‚(:“ ÌÔëÒµ—±“ gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚233.803' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.760600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N£„[ÆÈ ‚(:» ÌÔëÒ“ŒŸ• disk.sda1.in‚0 network.sent‚ 5.00378e+06 network.recv‚758889 disk.dm-0.in‚ 6231.90234375 disk.drbd2165.in‚0) disk./.usagePercent‚39.89223373086456 disk.sda.out‚ 74.54296875 disk.rbd1.in‚0 disk.rbd1.out‚ 7.08203125 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚594.4231109619141% disk.nvme0n1.out‚8720.87744140625 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.627860069274902 cpu‚26.134528794293196 disk.dm-1.out‚ 74.54296875$ memory_percent‚95.47624588012695 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 74.54296875# disk.nvme0c0n1.in‚ 6231.90234375& proc.memory.availableMB‚ 741.171875' disk.nvme0c0n1.out‚8720.87744140625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2724.30859375" disk.dm-0.out‚8720.87744140625 disk.sda.in‚0! disk.nvme0n1.in‚ 6231.90234375«u5à2É - _wandbruntime‚2777  _runtime‚2777dÎËwÇÊ Á _step‚264 & lr-AdamW/pg1‚0.0004260692489351527 & lr-AdamW/pg2‚0.0004260692489351527 trainer/global_step‚3249 % _timestamp‚1.7843431192629302e+09 _runtime‚2777.957398684õúe(ÌË Æ & lr-AdamW/pg1‚0.0004260692489351527 & lr-AdamW/pg2‚0.0004260692489351527 trainer/global_step‚3249 % _timestamp‚1.7843431192629302e+09 _runtime‚2777.957398684 _step‚264ˆ7ìž?2Ì - _wandbruntime‚2777  _runtime‚2777ıNâà'Í ÚO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497443726081536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5697291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6994730 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475886397952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3437664252416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881933 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3475977072128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091469 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12691753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55871954 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664252416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370717963776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367703040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367703040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55871954 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664252416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370717963776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367703040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367703040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12695551 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018225664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494152961519616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3439018225664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30117809 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890791 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117809 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12695827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676879360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898631 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497591979745280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6996791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112405167401984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494152961519616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843431198793144e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107818 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695193 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112405060219904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476909181952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108928151037952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚494152944480256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979745280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476997751808 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621359616ntÕï&Ñ éM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5699036 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6996518 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚491610783784960 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676879360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898631 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497591979745280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6996791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112405167401984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494152961519616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843431198793144e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107818 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695193 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112405060219904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476909181952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108928151037952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚494152944480256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979745280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476997751808 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621359616 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12695551 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018225664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494152961519616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3439018225664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30117809 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890791 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117809 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12695827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502272826245705 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30553218 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3488701093376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527153756672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3488359754240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114093313962496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12883522 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843431591614232e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563195 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501367754932224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚2817.855281847 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30563195 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 trainer/global_step‚3299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114686464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3483377128267 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3527240236544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30553218 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701093376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501367754932224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504856114686464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7099240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473064960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110566160205824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502273183834955 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504856473064960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7098984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 VFDeviceStatsMonito”cëwV r.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114686464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3483377128267 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3527240236544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30553218 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701093376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501367754932224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504856114686464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7099240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473064960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110566160205824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502273183834955 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504856473064960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7098984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502272826245705 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30553218 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3488701093376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527153756672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3488359754240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114093313962496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12883522 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843431591614232e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563195 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501367754932224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚2817.855281847 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30563195³×9v2ß - _wandbruntime‚2818  _runtime‚2818\KÅàê&à äM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30572261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110599588964352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3489372381184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7101028 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3484047632099 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12887311 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3528170486784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560661 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3489713727488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724690 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12887588 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚270 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30572261 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182101 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3528263250944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26162422 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726728704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7101302 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127870827520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505004368343040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713727488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562270 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114127759451136 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724690 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12887588 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚270 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30572261 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182101 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3528263250944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26162422 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726728704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7101302 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127870827520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505004368343040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713727488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562270 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114127759451136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110599588964352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3489372381184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7101028 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3484047632099 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12887311 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3528170486784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560661 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3489713727488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PÖÌí Æ _step‚272 & lr-AdamW/pg1‚0.0004244564531145746 & lr-AdamW/pg2‚0.0004244564531145746 trainer/global_step‚3349 % _timestamp‚1.7843431992552688e+09 _runtime‚2857.948826785i6e2î - _wandbruntime‚2857  _runtime‚2857‰YÛ á'ï ÛO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815966071296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843431992563398e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861669888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578471196160 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509648976326213 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017647 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115816087940608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7203952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚508729823453184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220048384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509649333915463 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13076011 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112237598132224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396595200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554037 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26536390 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396595200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506115323260928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5871791 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚3349 _runtime‚2857.950133943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544058 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7204220 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13075740 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508729823453184qÝGôæ'ð àO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861669888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578471196160 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509648976326213 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017647 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115816087940608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7203952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚508729823453184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220048384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509649333915463 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13076011 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112237598132224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396595200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554037 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26536390 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396595200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506115323260928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5871791 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚3349 _runtime‚2857.950133943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544058 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7204220 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13075740 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508729823453184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815966071296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843431992563398e+09‘A+wÞ2ñ - _wandbruntime‚2858  _runtime‚2858,Ñ*ñò äM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7206044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115850526204928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560742 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚508877047443456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579480701952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473712128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7206324 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409229312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067883008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚2858.565330312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚506261850972160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13079860 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13079577 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚508877064482816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843431998713331e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115326464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051807 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26544022 trainer/global_step‚3349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026713 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÿüGšï&ó éM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚2858.565330312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚506261850972160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13079860 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13079577 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚508877064482816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843431998713331e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115326464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051807 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26544022 trainer/global_step‚3349 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026713 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7206044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115850526204928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560742 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚508877047443456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579480701952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473712128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7206324 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409229312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067883008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438’ݲÜ2ô - _wandbruntime‚2858  _runtime‚2858ûÁœÀõ ‚ 2loss/FBt2/assignment_loss‚0.010703315081015774 1loss/SRqqt2/detection_loss‚0.6300944652144403 epoch‚8 trainer/global_step‚3349 $ _timestamp‚1.784343199873854e+09 0loss/FRt1/assignment_loss‚1.7947777692307818 2loss/SRqqt2/assignment_loss‚0.5199385495390869 /loss/FRt2/detection_loss‚1.2411049497490518 /loss/FBt1/detection_loss‚0.7413627977573103 /loss/FBt2/detection_loss‚0.6123822214789688 0loss/FRt2/assignment_loss‚0.9441715400856858 2loss/FBt1/assignment_loss‚0.013722261654544538 /loss/FRt1/detection_loss‚0.5938417666363169 %loss/total_loss‚8.848421773659243 1loss/SRqqt1/detection_loss‚1.0794391190126669 _runtime‚2858.567214408 _step‚275 2loss/SRqqt1/assignment_loss‚0.6668830182193728ÝQ¦ ö ‡ 0loss/FRt2/assignment_loss‚0.9441715400856858 2loss/FBt1/assignment_loss‚0.013722261654544538 /loss/FRt1/detection_loss‚0.5938417666363169 %loss/total_loss‚8.848421773659243 1loss/SRqqt1/detection_loss‚1.0794391190126669 _runtime‚2858.567214408 _step‚275 2loss/SRqqt1/assignment_loss‚0.6668830182193728 2loss/FBt2/assignment_loss‚0.010703315081015774 1loss/SRqqt2/detection_loss‚0.6300944652144403 epoch‚8 trainer/global_step‚3349 $ _timestamp‚1.784343199873854e+09 0loss/FRt1/assignment_loss‚1.7947777692307818 2loss/SRqqt2/assignment_loss‚0.5199385495390869 /loss/FRt2/detection_loss‚1.2411049497490518 /loss/FBt1/detection_loss‚0.7413627977573103 /loss/FBt2/detection_loss‚0.6123822214789688“(ïµ:“÷ ‚(:ˆ ¦ÕëÒžèë“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚187.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ñägÂø ‚(:· ¦ÕëÒú›— network.sent‚ 5.162514e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6446.2265625" disk.dm-0.out‚8763.38232421875 disk.sda1.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚8763.38232421875* proc.memory.percent‚16.627860069274902 cpu‚26.1212379504075 disk.sda2.in‚0% disk.nvme0n1.out‚8763.38232421875 disk.sda3.in‚0 disk.rbd1.out‚ 7.26953125# proc.memory.rssMB‚ 2724.30859375' proc.memory.availableMB‚ 740.8671875# disk./.usageGB‚594.423469543457 disk.sda1.in‚0 disk.sda.out‚ 75.91796875 disk.sda3.out‚ 75.91796875 proc.cpu.threads‚17 network.recv‚781380) disk./.usagePercent‚39.89225779557278 disk.rbd0.out‚0 disk.dm-1.out‚ 75.91796875" disk.nvme0c0n1.in‚ 6446.2265625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 6446.2265625$ memory_percent‚95.47810554504395›ÛÊ´žù ‚(:“ µÕëÒªÃë“ gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë/ã/³ú ‚(:¨ µÕëÒ€ˆî–) disk./.usagePercent‚39.89221427429196 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 494.7265625 network.sent‚ 5.212286e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 6446.2265625 cpu‚26.119156705148143 proc.cpu.threads‚17 network.recv‚785912' disk.nvme0c0n1.out‚8770.18115234375 disk.rbd0.out‚0" disk.dm-0.out‚8770.18115234375 disk.sda.out‚76.25 disk.rbd1.out‚ 7.39453125" disk.nvme0c0n1.in‚ 6446.2265625 disk.sda3.out‚76.25 disk.dm-1.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.98042869567871$ disk./.usageGB‚594.4228210449219 disk.sda3.in‚0 disk.dm-1.out‚76.25 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.627860069274902% disk.nvme0n1.out‚8770.18115234375 disk.dm-0.in‚ 6446.2265625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2724.30859375ñÁû ‚(:… ÄÕëÒ´¨ª“ gpu.0.gpu‚11.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¶ß”Èü ‚(:½ ÄÕëÒįþ–% disk.nvme0n1.out‚8782.71533203125 disk.sda3.in‚0 disk.sda3.out‚ 76.50390625( proc.memory.availableMB‚ 495.04296875 network.recv‚789156$ disk./.usageGB‚594.4228096008301 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 76.50390625! disk.nvme0n1.in‚ 6447.19140625 disk.sda1.in‚0 network.sent‚ 5.217219e+06 disk.dm-0.in‚ 6447.19140625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 cpu‚26.11708964977259# disk.nvme0c0n1.in‚ 6447.19140625' disk.nvme0c0n1.out‚8782.71533203125 disk.rbd0.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2724.30859375* proc.memory.percent‚16.627860069274902" disk.dm-0.out‚8782.71533203125 disk.rbd1.out‚ 7.39453125 disk.dm-1.in‚0 disk.dm-1.out‚ 76.50390625) disk./.usagePercent‚39.89221350626936$ memory_percent‚96.97849750518799 proc.cpu.threads‚17²Ý´2ý - _wandbruntime‚2898  _runtime‚2898ç¨Çþ Á trainer/global_step‚3399 % _timestamp‚1.7843432393200135e+09 _runtime‚2898.013586901 _step‚276 & lr-AdamW/pg1‚0.0004236307685629644 & lr-AdamW/pg2‚0.0004236307685629644 dfuÌÿ Æ % _timestamp‚1.7843432393200135e+09 _runtime‚2898.013586901 _step‚276 & lr-AdamW/pg1‚0.0004236307685629644 & lr-AdamW/pg2‚0.0004236307685629644 trainer/global_step‚3399”jžñ2€- _wandbruntime‚2898  _runtime‚2898šý„*à'ÚO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519681608653312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5959038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117538482561024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26917988 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113909054670848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103257088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚516091857895424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517025483995971 _runtime‚2898.014862559 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3590092097024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31462122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3399 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608653312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629446502400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390089 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚277 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519681967031808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472099 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31462122 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3399 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608653312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629446502400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390089 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚277 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519681967031808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472099 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516239115964416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7309861 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406787 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155917 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519829862309888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13270927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585280605915 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471174 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519830220695552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763384832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481165 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220695552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13270644 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104731136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117573053385216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516239115964416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7309861 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406787 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155917 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519829862309888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13270927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585280605915 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471174 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519830220695552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763384832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481165 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220695552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13270644 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104731136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117573053385216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355636736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚527094714015232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7411939 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3635226588991 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926551 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787598848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787598848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119261042714112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚527094355636736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714015232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13458230 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13457964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119260925039104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680451054080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784343279449675e+09 _step‚281 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926551 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787598848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787598848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119261042714112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚527094355636736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714015232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13458230 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13457964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119260925039104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680451054080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784343279449675e+09 _step‚281 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355636736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008™y/‹2“- _wandbruntime‚2938  _runtime‚2938+¢c"ê&”äM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3641458886656 trainer/global_step‚3449 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3641800232960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27307220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31935617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2938.758966832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523601150406656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925626 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681473838080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967678976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13461732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843432800652046e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925626 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681473838080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967678976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_poR3…h¹ol.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13461732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843432800652046e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2938.758966832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523601150406656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156987 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚530815960858624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120983552553984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983432781824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚534507460998656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32371026 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 % _timestamp‚1.7843433194854097e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731520870400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚530815977897984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507460998656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483100672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2978.179136899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381003 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102620160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567737 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0²ï³p·!¤àO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13650151 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3691483100672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3691141761536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141761536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32381003 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156987 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚530815960858624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120983552553984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983432781824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚534507460998656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32371026 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 % _timestamp‚1.7843433194854097e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731520870400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚530815977897984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507460998656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483100672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2978.179136899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381003 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102620160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceSx¯ Ò/tatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567737 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534507102620160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32371026 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6133541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3731622030336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528091924041728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7516610 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7516332 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0èbŸ2¥- _wandbruntime‚2978  _runtime‚2978Žqönê&¦äM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495734784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13653691 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3732555394560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 % _timestamp‚1.7843433201001706e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530963218927616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078891 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714662400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714662400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7518408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068898 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚3499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121018001149440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882693 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13653946 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117285359282176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2978.793937545 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154388480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚286 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27688822 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32380078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495734784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0æ’ ­ï&§éM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154388480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚286 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27688822 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32380078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495734784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495734784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13653691 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3732555394560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 % _timestamp‚1.7843433201001706e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530963218927616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078891 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714662400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714662400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7518408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068898 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚3499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121018001149440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882693 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13653946 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117285359282176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2978.793937545žj–Ñá2¨- _wandbruntime‚2978  _runtime‚2978Š™Å&©ÿ 2loss/FBt1/assignment_loss‚0.037284621245100534 .loss/FRt1/detection_loss‚0.614903496264383 %loss/total_loss‚9.000928574378683 _runtime‚2978.796347424 0loss/FRt1/assignment_loss‚1.9194733599382163 2loss/SRqqt1/assignment_loss‚0.6919942409765192 /loss/FRt2/detection_loss‚1.2228345747255454 .loss/FBt1/detection_loss‚0.583368048692743 epoch‚8 0loss/FRt2/assignment_loss‚1.0116856583527156 1loss/SRqqt1/detection_loss‚1.0491926584753841 /loss/FBt2/detection_loss‚0.5853332019436898 _step‚287 1loss/SRqqt2/assignment_loss‚0.605443119212491 1loss/FBt2/assignment_loss‚0.02891193902427489 1loss/SRqqt2/detection_loss‚0.6505036555276207 trainer/global_step‚3499 % _timestamp‚1.7843433201030517e+09¿?Ö€ ª„ trainer/global_step‚3499 % _timestamp‚1.7843433201030517e+09 2loss/FBt1/assignment_loss‚0.037284621245100534 .loss/FRt1/detection_loss‚0.614903496264383 %loss/total_loss‚9.000928574378683 _runtime‚2978.796347424 0loss/FRt1/assignment_loss‚1.9194733599382163 2loss/SRqqt1/assignment_loss‚0.6919942409765192 /loss/FRt2/detection_loss‚1.2228345747255454 .loss/FBt1/detection_loss‚0.583368048692743 epoch‚8 0loss/FRt2/assignment_loss‚1.0116856583527156 1loss/SRqqt1/detection_loss‚1.0491926584753841 /loss/FBt2/detection_loss‚0.5853332019436898 _step‚287 1loss/SRqqt2/assignment_loss‚0.605443119212491 1loss/FBt2/assignment_loss‚0.02891193902427489 1loss/SRqqt2/detection_loss‚0.6505036555276207ŸŒæ«û’«‚(:‡ žÖëÒêë“” gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q‘²§¼¬‚(:± žÖëÒ×îË• network.recv‚811382 disk.dm-1.in‚0 disk.rbd1.out‚ 7.64453125 disk.rbd0.out‚0$ disk./.usageGB‚594.4229164123535 disk.sda.out‚ 82.2734375 disk.sda2.out‚0 disk.nvme0n1.in‚ 6686.1484375 disk.rbd0.in‚0 cpu‚26.104907796059827) disk./.usagePercent‚39.89222067448031 disk.dm-0.out‚8835.384765625# proc.memory.rssMB‚ 2724.30859375 disk.sda3.in‚0 disk.sda3.out‚ 82.2734375$ memory_percent‚96.98026180267334 proc.cpu.threads‚17 disk.dm-1.out‚ 82.2734375" disk.nvme0c0n1.in‚ 6686.1484375% disk.nvme0c0n1.out‚8835.384765625 disk.dm-0.in‚ 6686.1484375# disk.nvme0n1.out‚8835.384765625 network.sent‚ 5.36282e+06 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 494.75390625* proc.memory.percent‚16.6278600692749023êEÿ‘­‚(:† ­ÖëÒ³ó¿“ gpu.0.gpu‚51.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õýéÈ®‚(:½ ­ÖëÒŸŒê• disk.sda2.in‚0$ memory_percent‚96.98078632354736 disk.sda.in‚0 disk.sda3.in‚0) disk./.usagePercent‚39.89221965045018 disk.rbd0.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6686.54296875$ disk./.usageGB‚594.4229011535645 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6686.54296875 disk.sda.out‚ 82.53515625 network.sent‚ 5.425658e+06 disk.sda1.in‚0! disk.nvme0n1.in‚ 6686.54296875% disk.nvme0n1.out‚8840.86083984375( proc.memory.availableMB‚ 494.66796875# proc.memory.rssMB‚ 2724.30859375 network.recv‚815926 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚8840.86083984375" disk.dm-0.out‚8840.86083984375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 82.53515625 disk.sda3.out‚ 82.53515625 cpu‚26.10298073433702 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 7.70703125* proc.memory.percent‚16.627860069274902+i&ž¯‚(:“ ¼ÖëÒØ«Ú” gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Ö—â¼°‚(:± ¼ÖëÒœ»’–' disk.nvme0c0n1.out‚8849.15771484375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚819014 disk.sda.out‚82.84375* proc.memory.percent‚16.627860069274902$ disk./.usageGB‚594.4229545593262 disk.dm-1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚8849.15771484375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚82.84375% disk.nvme0n1.out‚8849.15771484375 disk.sda3.out‚82.84375 cpu‚26.10098911076401) disk./.usagePercent‚39.89222323455566 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 6686.98046875 disk.sda2.out‚0 disk.rbd1.out‚ 7.70703125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 6686.98046875 disk.sda.in‚0% proc.memory.availableMB‚ 494.90625# proc.memory.rssMB‚ 2724.30859375 network.sent‚ 5.430687e+06 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 6686.98046875$ memory_percent‚96.97933197021484¤^Ãu2±- _wandbruntime‚3018  _runtime‚30180PÊÔɲà ' lr-AdamW/pg1‚0.00042107720718260626 ' lr-AdamW/pg2‚0.00042107720718260626 trainer/global_step‚3549 % _timestamp‚1.7843433594039567e+09 _runtime‚3018.097527229 _step‚288˜EšÎ³È ' lr-AdamW/pg1‚0.00042107720718260626 ' lr-AdamW/pg2‚0.00042107720718260626 trainer/global_step‚3549 % _timestamp‚1.7843433594039567e+09 _runtime‚3018.097527229 _step‚288 8Çhª2´- _wandbruntime‚3018  _runtime‚3018²Ú µÚO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6220788 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7621288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541919849603584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825478 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118923368450048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782665640448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3736119012917 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837263360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782570771968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7621011 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.currenX5C`Õt‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3736459562807 % _timestamp‚1.7843433594050398e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178602496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32835455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849603584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3742178602496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚535417457635328 trainer/global_step‚3549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538178012340224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122706034090496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648245 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207982080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207982080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13842079 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13841799 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825478 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122705920609792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118923349837824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3018.098776026 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7621288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541919849603584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825478 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118923368450048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782665640448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3736119012917 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837263360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782570771968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7621011 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3736459562807 % _timestamp‚1.7843433594050398e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178602496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32835455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849603584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3742178602496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚535417457635328 trainer/global_step‚3549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538178012340224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122706034090496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648245 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207982080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207982080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13842079 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13841799 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825478 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122705920609792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118923349837824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3018.098776026 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13845620 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7623339 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7623087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118956797208576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542068461645824 _step‚290 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103260160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783587020800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13845875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904950 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538325270409216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3742849890304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963201 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616641 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6222533 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538325253369856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169XÆW¹éM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914943 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚535563968307200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28070422 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13845620 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7623339 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7623087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118956797208576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542068461645824 _step‚290 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103260160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783587020800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13845875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904950 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538325270409216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3742849890304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963201 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616641 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6222533 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538325253369856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3743191236608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122740365617152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2302517657ØMYiè 6 _runtime‚3018.713110034 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740476993536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E” ¿¾‚(:´ ËÖëÒÑϦ• disk.sda1.in‚0 disk.dm-1.out‚ 83.046875 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 6688.76171875$ disk.nvme0n1.out‚8863.4267578125 disk.sda1.out‚0 disk.sda3.in‚0* proc.memory.percent‚16.627860069274902 network.sent‚ 5.435513e+06! disk.nvme0n1.in‚ 6688.76171875# proc.memory.rssMB‚ 2724.30859375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 83.046875 disk.rbd1.out‚ 7.76953125$ memory_percent‚96.98078632354736( disk./.usagePercent‚39.8922810922584 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8863.4267578125 disk.dm-0.in‚ 6688.76171875 disk.sda.in‚0 disk.sda.out‚ 83.046875 disk.rbd1.in‚0 network.recv‚822378$ disk./.usageGB‚594.4238166809082 disk.dm-1.in‚0! disk.dm-0.out‚8863.4267578125( proc.memory.availableMB‚ 494.66796875 cpu‚26.099037748856702 proc.cpu.threads‚17õñΪž¿‚(:“ ÚÖëÒ“Å–” gpu.0.gpu‚70.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸¥ÙW¼À‚(:± ÚÖëÒè¡ÿ• disk.drbd2165.out‚0 disk.rbd1.out‚ 7.76953125& disk.nvme0c0n1.out‚8866.6396484375* disk./.usagePercent‚39.892278532183056 disk.dm-0.in‚ 6801.0859375! disk.dm-0.out‚8866.6396484375 disk.sda.in‚0$ disk./.usageGB‚594.4237785339355 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 6801.0859375$ disk.nvme0n1.out‚8866.6396484375* proc.memory.percent‚16.627860069274902 network.sent‚ 5.498602e+06$ memory_percent‚96.98076248168945 proc.cpu.threads‚17& proc.memory.availableMB‚ 494.671875 disk.dm-1.in‚0 network.recv‚827075 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚26.097240814291453 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 6801.0859375 disk.sda3.out‚ 83.234375# proc.memory.rssMB‚ 2724.30859375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 83.234375 disk.sda.out‚ 83.234375ꆋ~2Á- _wandbruntime‚3058  _runtime‚3058}=KÉÂà _step‚292 ' lr-AdamW/pg1‚0.00042020069971489645 ' lr-AdamW/pg2‚0.00042020069971489645 trainer/global_step‚3599 % _timestamp‚1.7843433994693956e+09 _runtime‚3058.162961843›¯‹×ÎÃÈ _runtime‚3058.162961843 _step‚292 ' lr-AdamW/pg1‚0.00042020069971489645 ' lr-AdamW/pg2‚0.00042020069971489645 trainer/global_step‚3599 % _timestamp‚1.7843433994693956e+09¤v ¡2Ä- _wandbruntime‚3058  _runtime‚3058šœâ´à'ÅÚO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954965504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14033662 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3786735499825 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596587008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3792874104320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚3058.164267798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532765184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874104320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚293 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28444388 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33279930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3833556522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049715 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784343399470489e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724318 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3833657682432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428344286720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28444390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124428464058880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33289907 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596587008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954965504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7725624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7725887 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532765184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14033928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728753 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976²Iíçå'ÆßO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724318 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3833657682432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428344286720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28444390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124428464058880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33289907 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596587008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954965504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7725624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7725887 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532765184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14033928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728753 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954965504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14033662 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3786735499825 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596587008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3792874104320 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚3058.164267798 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532765184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874104320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚293 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28444388 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33279930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3833556522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049715 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784343399470489e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591¥h¹ë2Ç- _wandbruntime‚3058  _runtime‚3058Ÿ‘‚ÒÆÈäM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61750995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33288982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043709 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741002 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14037381 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549481208629248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ÁnÝc$J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549480850243584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14037661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚546677248454347 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542889501900800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚3058.778595869 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3834673880576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28452022 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545687321890816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545687304851456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3599 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549481208629248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298973 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33288982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120628235134976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793886738432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¡ø)¬ï&ÉéM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61750995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33288982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043709 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741002 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14037381 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549481208629248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549480850243584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14037661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚546677248454347 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542889501900800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚3058.778595869 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3834673880576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28452022 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545687321890816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545687304851456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3599 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549481208629248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298973 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33288982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120628235134976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793886738432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360¦€œõ•2Ê- _wandbruntime‚3058  _runtime‚3058­Ï¹Ë€ 0loss/FRt2/assignment_loss‚1.1331183530175637 2loss/SRqqt2/assignment_loss‚0.5263307765930567 1loss/FBt1/assignment_loss‚0.01088560334853676 /loss/FRt1/detection_loss‚0.5621482693784369 1loss/SRqqt1/detection_loss‚1.0228958070160938 %loss/total_loss‚9.259104984358665 % _timestamp‚1.7843434000872931e+09 _step‚295 0loss/FRt1/assignment_loss‚1.7629485823498885 /loss/FRt2/detection_loss‚1.4248539693628433 epoch‚8 1loss/FBt2/assignment_loss‚0.13033351637287469 /loss/FBt1/detection_loss‚0.7939055658986464 1loss/SRqqt1/assignment_loss‚0.613727788038225 1loss/SRqqt2/detection_loss‚0.6408256282194542 /loss/FBt2/detection_loss‚0.6371311247630436 trainer/global_step‚3599 _runtime‚3058.780612867Œ×» Ì… 0loss/FRt1/assignment_loss‚1.7629485823498885 /loss/FRt2/detection_loss‚1.4248539693628433 epoch‚8 1loss/FBt2/assignment_loss‚0.13033351637287469 /loss/FBt1/detection_loss‚0.7939055658986464 1loss/SRqqt1/assignment_loss‚0.613727788038225 1loss/SRqqt2/detection_loss‚0.6408256282194542 /loss/FBt2/detection_loss‚0.6371311247630436 trainer/global_step‚3599 _runtime‚3058.780612867 0loss/FRt2/assignment_loss‚1.1331183530175637 2loss/SRqqt2/assignment_loss‚0.5263307765930567 1loss/FBt1/assignment_loss‚0.01088560334853676 /loss/FRt1/detection_loss‚0.5621482693784369 1loss/SRqqt1/detection_loss‚1.0228958070160938 %loss/total_loss‚9.259104984358665 % _timestamp‚1.7843434000872931e+09 _step‚295§3R÷œÍ‚(:‘ éÖëÒ‹«³” gpu.0.gpu‚17.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.516' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.30320000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã;¶ÚÁ΂(:¶ éÖëÒÕ‚ª˜! disk.dm-0.out‚8877.9736328125 disk.sda3.out‚ 83.5234375$ memory_percent‚96.98071479797363 disk.rbd0.in‚0 network.sent‚ 5.516625e+06 network.recv‚830497# disk./.usageGB‚594.423469543457 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 83.5234375 disk.sda2.out‚0! disk.nvme0n1.in‚ 6801.91796875 cpu‚26.096484488297275 proc.cpu.threads‚17 disk.dm-1.out‚ 83.5234375# disk.nvme0c0n1.in‚ 6801.91796875 disk.sda2.in‚0 disk.sda3.in‚0) disk./.usagePercent‚39.89225779557278 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚8877.9736328125$ disk.nvme0n1.out‚8877.9736328125 disk.rbd1.in‚0 disk.rbd1.out‚ 7.83203125# proc.memory.rssMB‚ 2724.30859375* proc.memory.percent‚16.627860069274902 disk.drbd2165.in‚0 disk.dm-0.in‚ 6801.91796875 disk.sda.in‚0' proc.memory.availableMB‚ 494.6796875ucžÏ‚(:“ øÖëҬ龓 gpu.0.gpu‚87.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚292.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.528400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n!ڿЂ(:´ øÖëÒŸÎÀ• proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚8885.5712890625 disk.dm-0.in‚ 6802.8828125 disk.sda.out‚ 83.92578125 disk.rbd1.out‚ 7.83203125# proc.memory.rssMB‚ 2728.26953125( proc.memory.percent‚16.6520357131958 cpu‚26.11322461482388$ disk./.usageGB‚594.4234275817871 disk.sda1.out‚0$ disk.nvme0n1.out‚8885.5712890625 disk.rbd1.in‚0 network.sent‚ 5.56639e+06 network.recv‚835025 disk.dm-1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 490.8515625 disk.dm-1.out‚ 83.92578125 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 6802.8828125* disk./.usagePercent‚39.892254979489906 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.00407981872559 disk.rbd0.out‚0! disk.dm-0.out‚8885.5712890625 disk.nvme0n1.in‚ 6802.8828125 disk.sda3.in‚0 disk.sda3.out‚ 83.92578125xC&ø2Ñ- _wandbruntime‚3075  _runtime‚3075pÿãq2ÒÈ{ 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8219247460365295 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.863000750541687 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0463738739490509 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1978721171617508 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9038443565368652 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6441876292228699 +jet/accuracy_1_of_1‚0.33017536997795105 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7054391503334045 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9744486808776855 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7568112015724182 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017723549157381058 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936426877975464 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8716753125190735 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07959580421447754 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9275342226028442 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23916834592819214 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871848225593567 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7904167175292969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6232455372810364 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9903203248977661 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_1_of_2‚0.779116690158844 6validation_average_jet_accuracy‚0.3699600100517273 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6985352039337158 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2102777361869812 *jet/accuracy_1_of_4‚0.9860848784446716 epoch‚8 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6491202712059021 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6504439115524292 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19416548311710358 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6564438343048096 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32040491700172424 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795319437980652 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08352728933095932 +particle/sensitivity‚0.6185930371284485 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973257184028625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4726769030094147 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029535315930843353 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7621738910675049 *jet/accuracy_2_of_3‚0.5402777194976807 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922181963920593 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5830302238464355 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.762584388256073 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.877839207649231 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0649353414773941 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7689040899276733 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6350250244140625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22970697283744812 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6636844873428345 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44412532448768616 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7661998271942139 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031511805951595306 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7719710469245911 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6294193267822266 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5116666555404663 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04883257672190666 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17478400468826294 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 +particle/specificity‚0.9051123261451721 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6893568634986877 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22951066493988037 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.877839207649231 *jet/accuracy_2_of_4‚0.7895709276199341 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7693625688552856 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02503924071788788 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5702673196792603 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6983548998832703 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39570799469947815 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.994326651096344 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6658774018287659 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.677150547504425 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 _runtime‚3075.930210969 _step‚296 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7694823145866394 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7696512341499329 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 (particle/accuracy‚0.8435795307159424 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9562169909477234 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22815348207950592 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02851371467113495 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7308619618415833 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967526793479919 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6441876292228699 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19595037400722504 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.286199152469635 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8605253100395203 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022567732259631157 /particle/accuracy_3_of_4‚0.9965986609458923 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28842490911483765 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3805767893791199 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08267746865749359 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24244645237922668 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8500944972038269 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8221210241317749 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7248339653015137 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4315250515937805 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1897456794977188 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333432674408 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7040718197822571 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20772945880889893 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_3‚0.9172417521476746 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6474243402481079 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473555445671082 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9829365015029907 ,jet/accuracy_4_of_4‚0.013899499550461769 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19524578750133514 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 +jet/accuracy_2_of_2‚0.14338473975658417 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 )loss/val_total_loss‚8.601547087662471 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03463365510106087 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.260966956615448 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.843272864818573 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 jet/accuracy_5_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5206300616264343 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843434172358875e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9769073724746704 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8414778113365173 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6467169523239136 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919400215148926 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06824375689029694 :"Pu+†um] rity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.663375973701477 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18154193460941315 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.719352662563324 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5057028532028198 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7674958109855652 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948029518127 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38072967529296875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17306849360466003 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592835426331 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9590639472007751 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19082580506801605 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7114254236221313 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8677681684494019 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6449838876724243 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9838080406188965 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18336151540279388 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24142029881477356 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027425825595855713 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18258719146251678 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4082060754299164 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47167328000068665 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022567732259631157 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22125418484210968 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10135876387357712 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909600019454956 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22970697283744812 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03715139999985695 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3599C61Ó=ÓÍ{ 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973257184028625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4726769030094147 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029535315930843353 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7621738910675049 *jet/accuracy_2_of_3‚0.5402777194976807 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922181963920593 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5830302238464355 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.762584388256073 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.877839207649231 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0649353414773941 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7689040899276733 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6350250244140625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22970697283744812 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6636844873428345 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44412532448768616 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7661998271942139 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031511805951595306 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7719710469245911 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6294193267822266 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5116666555404663 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04883257672190666 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17478400468826294 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 +particle/specificity‚0.9051123261451721 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6893568634986877 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22951066493988037 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.877839207649231 *jet/accuracy_2_of_4‚0.7895709276199341 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7693625688552856 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02503924071788788 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5702673196792603 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6983548998832703 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39570799469947815 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.994326651096344 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6658774018287659 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.677150547504425 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 _runtime‚3075.930210969 _step‚296 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7694823145866394 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7696512341499329 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 (particle/accuracy‚0.8435795307159424 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9562169909477234 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22815348207950592 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02851371467113495 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7308619618415833 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967526793479919 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6441876292228699 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19595037400722504 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.286199152469635 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8605253100395203 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022567732259631157 /particle/accuracy_3_of_4‚0.9965986609458923 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28842490911483765 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3805767893791199 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08267746865749359 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24244645237922668 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8500944972038269 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8221210241317749 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7248339653015137 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4315250515937805 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1897456794977188 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333432674408 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7040718197822571 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20772945880889893 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_3‚0.9172417521476746 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6474243402481079 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473555445671082 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9829365015029907 ,jet/accuracy_4_of_4‚0.013899499550461769 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19524578750133514 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 +jet/accuracy_2_of_2‚0.14338473975658417 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 )loss/val_total_loss‚8.601547087662471 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03463365510106087 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.260966956615448 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.843272864818573 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 jet/accuracy_5_of_5‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5206300616264343 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843434172358875e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9769073724746704 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8414778113365173 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6467169523239136 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919400215148926 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06824375689029694 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.663375973701477 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18154193460941315 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.719352662563324 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5057028532028198 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7674958109855652 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948029518127 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38072967529296875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17306849360466003 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592835426331 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9590639472007751 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19082580506801605 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7114254236221313 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8677681684494019 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6449838876724243 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9838080406188965 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18336151540279388 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24142029881477356 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027425825595855713 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18258719146251678 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4082060754299164 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47167328000068665 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022567732259631157 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22125418484210968 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10135876387357712 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909600019454956 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22970697283744812 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03715139999985695 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3599 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8219247460365295 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.863000750541687 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0463738739490509 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1978721171617508 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9038443565368652 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6441876292228699 +jet/accuracy_1_of_1‚0.33017536997795105 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7054391503334045 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9744486808776855 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7568112015724182 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017723549157381058 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936426877975464 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8716753125190735 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07959580421447754 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9275342226028442 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23916834592819214 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871848225593567 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7904167175292969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6232455372810364 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9903203248977661 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_1_of_2‚0.779116690158844 6validation_average_jet_accuracy‚0.3699600100517273 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6985352039337158 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2102777361869812 *jet/accuracy_1_of_4‚0.9860848784446716 epoch‚8 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6491202712059021 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6504439115524292 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19416548311710358 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6564438343048096 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32040491700172424 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795319437980652 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08352728933095932 +particle/sensitivity‚0.6185930371284485¨Vz|i“Ô‚(:ˆ ‡×ëÒƒ¢Ö• gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÙ¶Õ‚(:« ‡×ëÒÀÒû– network.recv‚838597 disk.sda3.out‚ 84.203125 cpu‚26.11361657386415$ disk./.usageGB‚594.4233665466309 disk.sda2.out‚0$ disk.nvme0n1.out‚8889.7021484375 disk.dm-1.out‚ 84.203125 disk.drbd2165.in‚0 disk.dm-0.in‚ 6802.8828125 disk.sda1.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 6802.8828125 network.sent‚ 5.571327e+06 disk.nvme0n1.in‚ 6802.8828125 disk.sda3.in‚0 disk.sda1.in‚0! disk.dm-0.out‚8889.7021484375$ proc.memory.availableMB‚490.9375" proc.memory.rssMB‚ 2728.2734375) proc.memory.percent‚16.65205955505371 disk.rbd1.out‚ 7.95703125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚8889.7021484375 disk.sda.in‚0$ memory_percent‚97.00355529785156 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 84.203125 disk.rbd0.out‚0) disk./.usagePercent‚39.89225088336936 disk.drbd2165.out‚0ÁysXžÖ‚(:“ –×ëÒе¾“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ì»×‚(:° –×ëÒò·ö– disk.sda1.in‚0! disk.nvme0c0n1.in‚ 6904.484375 disk.sda.out‚ 84.3984375 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2728.2734375 network.sent‚ 5.628837e+06 disk.dm-0.in‚ 6904.484375$ disk.nvme0n1.out‚8896.9990234375 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 84.3984375# memory_percent‚97.0050573348999 disk.sda3.out‚ 84.3984375) proc.memory.percent‚16.65205955505371 network.recv‚842901 disk.drbd2165.in‚0* disk./.usagePercent‚39.892287492446755$ disk./.usageGB‚594.4239120483398! disk.dm-0.out‚8896.9990234375 disk.rbd1.in‚0 disk.rbd0.in‚0 cpu‚26.111637291649775 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd1.out‚ 7.95703125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚8896.9990234375( proc.memory.availableMB‚ 490.69140625 disk.nvme0n1.in‚ 6904.484375­`2Ø- _wandbruntime‚3115  _runtime‚3115O²'zÇÙÁ & lr-AdamW/pg1‚0.0004193116252278821 & lr-AdamW/pg2‚0.0004193116252278821 trainer/global_step‚3649 % _timestamp‚1.7843434567900078e+09 _runtime‚3115.483591601 _step‚297‹ä›ÌÚÆ _step‚297 & lr-AdamW/pg1‚0.0004193116252278821 & lr-AdamW/pg2‚0.0004193116252278821 trainer/global_step‚3649 % _timestamp‚1.7843434567900078e+09 _runtime‚3115.483591601©¶Ú*2Û- _wandbruntime‚3115  _runtime‚3115üÝqß'ÜÙO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751358 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14257857 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554831681953792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚298 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7860046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643426304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558693001804800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843434567911007e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643426304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28946654 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122282156402688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185130665984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 _runtime‚3115.484832254 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261852 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.©8Hà%on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558693001804800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843434567911007e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643426304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28946654 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122282156402688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185130665984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 _runtime‚3115.484832254 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261852 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14257857 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554831681953792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚298 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7860046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643426304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ªZ_†Ã2Þ- _wandbruntime‚3116  _runtime‚3116(Öê&ßäM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778035 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315445760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554978922983424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126219472889344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3903997007360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3903906340352 _step‚299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255468544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315686 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861974099456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843434574062502e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768042 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14261361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33813756 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7861808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28954288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974099456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315445760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122315566548992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ê’ï&àéM trainer/global_step‚3649 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778035 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315445760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554978922983424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126219472889344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3903997007360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3903906340352 _step‚299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255468544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315686 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861974099456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843434574062502e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768042 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14261361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33813756 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7861808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28954288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974099456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315445760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122315566548992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 Bë¼Æå‚(:» ¥×ëÒÝð˜ disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 84.75390625" disk.dm-0.out‚8907.76708984375 network.recv‚846512$ disk./.usageGB‚594.4239921569824 disk.dm-0.in‚ 6905.44921875 disk.sda3.in‚0 network.sent‚ 5.633858e+06 disk.rbd0.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 6905.44921875% disk.nvme0n1.out‚8907.76708984375 disk.sda3.out‚ 84.75390625 proc.cpu.threads‚17* disk./.usagePercent‚39.892292868604976 disk.drbd2165.out‚0$ memory_percent‚97.00605869293213( proc.memory.availableMB‚ 490.52734375 disk.rbd1.out‚ 8.01953125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6905.44921875 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2728.2734375' disk.nvme0c0n1.out‚8907.76708984375 disk.sda.out‚ 84.75390625 disk.dm-1.in‚0 cpu‚26.1098144465664 disk.sda1.out‚0) proc.memory.percent‚16.65205955505371Ö¤ °&æ‚(:ˆ ´×ëÒñ¾²• gpu.0.gpuɉÚMm‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚229.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\MÍ8©ç‚(:ž ´×ëҀɷ–" proc.memory.rssMB‚ 2728.2734375 disk.dm-1.out‚85& disk.nvme0c0n1.out‚8912.0029296875 disk.dm-0.in‚ 6905.44921875 disk.rbd1.in‚0 cpu‚26.107900536609392$ disk./.usageGB‚594.4235992431641 disk.rbd0.out‚0$ disk.nvme0n1.out‚8912.0029296875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 8.01953125( proc.memory.availableMB‚ 490.83984375 network.sent‚ 5.697814e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.65205955505371 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6905.44921875! disk.dm-0.out‚8912.0029296875 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚849974 disk.sda3.out‚85) disk./.usagePercent‚39.89226649982894 disk.rbd0.in‚0 disk.sda.out‚85$ memory_percent‚97.00415134429932! disk.nvme0n1.in‚ 6905.44921875 disk.sda3.in‚03ôû‘“è‚(:ˆ Ã×ëÒ¿ƒ·“ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äW?b·é‚(:¬ Ã×ëÒš¨‘– disk.sda.out‚ 85.3046875 cpu‚26.10613819866606 network.sent‚ 5.700837e+06 disk.dm-1.out‚ 85.3046875' disk.nvme0c0n1.out‚8923.19580078125 disk.dm-0.in‚ 6906.28125 disk.rbd1.out‚ 8.08203125 disk.sda3.out‚ 85.3046875) proc.memory.percent‚16.65205955505371' disk./.usagePercent‚39.892300548831$ disk./.usageGB‚594.4241065979004 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 490.73046875 network.recv‚853524 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 6906.28125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2728.2734375" disk.dm-0.out‚8923.19580078125 disk.sda2.out‚0 disk.rbd0.in‚0# memory_percent‚97.0048189163208 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 6906.28125% disk.nvme0n1.out‚8923.19580078125 disk.sda3.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.in‚0ÜVñ¡2ê- _wandbruntime‚3155  _runtime‚3155‡FRþÉëà % _timestamp‚1.7843434967449858e+09 _runtime‚3155.438504323 _step‚301 ' lr-AdamW/pg1‚0.00041841004082515616 ' lr-AdamW/pg2‚0.00041841004082515616 trainer/global_step‚369992·ÎìÈ ' lr-AdamW/pg1‚0.00041841004082515616 ' lr-AdamW/pg2‚0.00041841004082515616 trainer/global_step‚3699 % _timestamp‚1.7843434967449858e+09 _runtime‚3155.438504323 _step‚301­'—›»2í- _wandbruntime‚3155  _runtime‚3155­NêÉà'îÚO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562193733435392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998313472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259156 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 trainer/global_step‚3699 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚563208767418328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390409728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748788224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587410 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198870 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954160151552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34269133 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 _runtime‚3155.439713225 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597389 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚566105748788224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390409728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123953575716864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6485061 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7962955 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656974336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911656974336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 (•å'ïßO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34269133 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 _runtime‚3155.439713225 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597389 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚566105748788224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390409728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123953575716864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6485061 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7962955 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656974336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911656974336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562193733435392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998313472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259156 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 trainer/global_step‚3699 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚563208767418328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390409728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748788224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587410 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198870 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954160151552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169®¨£Èk2ð- _wandbruntime‚3156  _runtime‚3156ãz6Z#ñäM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3955274913280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3912669601280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14451765 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002451968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34268208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚563355931554672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843434973615828e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562340991504384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚559449232302080 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7964959 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604094 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002451968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29335886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚3699 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010947584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚566253644066304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127942298000896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3955190537728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚563356289144034 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942195013120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3907039802594 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559449249341440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34268208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14451513 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604094 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 G:DeviceStatsMonitor.on_train_batch_end/active‰Y¦P.all.allocated‚63614087 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3913010947584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127942298000896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3955190537728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚563356289144034 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942195013120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3907039802594 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559449249341440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34268208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14451513 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604094 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63614087 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3913010947584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3955274913280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3912669601280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14451765 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002451968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34268208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚563355931554672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843434973615828e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562340991504384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚559449232302080 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7964959 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604094 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002451968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29335886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088¯ÜÔQ!2ó- _wandbruntime‚3156  _runtime‚3156ºŠ#Þô 2loss/SRqqt2/assignment_loss‚0.5096416599838585 1loss/SRqqt1/detection_loss‚0.9429559368673446 0loss/FBt2/detection_loss‚0.47304272890861077 /loss/FRt1/detection_loss‚0.5778744547819642 /loss/FBt1/detection_loss‚0.7020067728503987 %loss/total_loss‚8.993672704867471 trainer/global_step‚3699 _step‚304 0loss/FRt2/assignment_loss‚1.0738218889488438 2loss/SRqqt1/assignment_loss‚0.6679223021444028 1loss/SRqqt2/detection_loss‚0.6183251797598504 _runtime‚3156.057541227 0loss/FRt1/assignment_loss‚1.9029698881037034 1loss/FBt1/assignment_loss‚0.15012572032931834 1loss/FBt2/assignment_loss‚0.03031771488756875 /loss/FRt2/detection_loss‚1.3446684573016061 epoch‚9 $ _timestamp‚1.784343497364247e+09T8os õ† trainer/global_step‚3699 _step‚304 0loss/FRt2/assignment_loss‚1.0738218889488438 2loss/SRqqt1/assignment_loss‚0.6679223021444028 1loss/SRqqt2/detection_loss‚0.6183251797598504 _runtime‚3156.057541227 0loss/FRt1/assignment_loss‚1.9029698881037034 1loss/FBt1/assignment_loss‚0.15012572032931834 1loss/FBt2/assignment_loss‚0.03031771488756875 /loss/FRt2/detection_loss‚1.3446684573016061 epoch‚9 $ _timestamp‚1.784343497364247e+09 2loss/SRqqt2/assignment_loss‚0.5096416599838585 1loss/SRqqt1/detection_loss‚0.9429559368673446 0loss/FBt2/detection_loss‚0.47304272890861077 /loss/FRt1/detection_loss‚0.5778744547819642 /loss/FBt1/detection_loss‚0.7020067728503987 %loss/total_loss‚8.993672704867471°øÊœžö‚(:“ Ò×ëÒÏÑ– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00F»÷‚(:° Ò×ëÒ긙™) disk./.usagePercent‚39.89229952480086$ disk./.usageGB‚594.4240913391113" disk.nvme0c0n1.in‚ 7026.9296875 disk.sda3.out‚ 85.453125' proc.memory.availableMB‚ 490.4453125" proc.memory.rssMB‚ 2728.2734375 proc.cpu.threads‚17 disk.sda.in‚0% disk.nvme0n1.out‚8932.75634765625 disk.rbd1.in‚0 network.recv‚856762 disk.sda1.out‚0 disk.dm-1.out‚ 85.453125 disk.sda.out‚ 85.453125) proc.memory.percent‚16.65205955505371 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 network.sent‚ 5.705786e+06' disk.nvme0c0n1.out‚8932.75634765625 disk.dm-0.in‚ 7026.9296875 disk.sda2.in‚0 disk.rbd1.out‚ 8.08203125 disk.dm-1.in‚0" disk.dm-0.out‚8932.75634765625 disk.sda2.out‚0 disk.nvme0n1.in‚ 7026.9296875$ memory_percent‚97.00655937194824 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚26.10409984450228“ÁU’ø‚(:‡ á×ëÒî郔 gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚261.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.201 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ÔkÀù‚(:µ á×ëÒ²ù£– disk.drbd2165.out‚0 disk.nvme0n1.in‚ 7026.9296875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2728.2734375) proc.memory.percent‚16.65205955505371 cpu‚26.10249501114928 disk.sda1.in‚0 disk.dm-0.in‚ 7026.9296875 disk.sda.in‚0$ disk./.usageGB‚594.4239501953125 disk.dm-1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚8939.89599609375 disk.sda2.in‚0 disk.dm-1.out‚ 85.6953125" disk.dm-0.out‚8939.89599609375 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.00567722320557 disk.rbd1.out‚ 8.14453125 disk.sda1.out‚0) disk./.usagePercent‚39.89229005252209 disk.sda.out‚ 85.6953125 disk.sda2.out‚0% disk.nvme0n1.out‚8939.89599609375 disk.sda3.in‚0 disk.sda3.out‚ 85.6953125 network.sent‚ 5.768633e+06 network.recv‚861422( proc.memory.availableMB‚ 490.58984375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 7026.9296875j²t~‘ú‚(:† ð×ëÒ¹þÁ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ì’¥Åû‚(:º ð×ëÒÕñù–) disk./.usagePercent‚39.89228902849196" disk.dm-0.out‚8949.04638671875! disk.nvme0n1.in‚ 7027.76171875% disk.nvme0n1.out‚8949.04638671875" proc.memory.rssMB‚ 2728.2734375) proc.memory.percent‚16.65205955505371 disk.dm-1.in‚0 disk.dm-1.out‚ 85.96484375 disk.dm-0.in‚ 7027.76171875 disk.rbd0.out‚0$ disk./.usageGB‚594.4239349365234 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 490.08984375 network.recv‚864746 disk.sda3.in‚0 disk.rbd1.out‚ 8.14453125 proc.cpu.threads‚17 network.sent‚ 5.773606e+06 disk.sda.in‚0 disk.sda.out‚ 85.96484375 disk.sda3.out‚ 85.96484375 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 7027.76171875' disk.nvme0c0n1.out‚8949.04638671875$ memory_percent‚97.00872898101807 cpu‚26.1006875337238 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0ˆ&Â72ü- _wandbruntime‚3195  _runtime‚3195WÕ¸¦ÇýÁ & lr-AdamW/pg1‚0.0004174960044138001 & lr-AdamW/pg2‚0.0004174960044138001 trainer/global_step‚3749 % _timestamp‚1.7843435367974575e+09 _runtime‚3195.491092713 _step‚305Ä| ÌþÆ _runtime‚3195.491092713 _step‚305 & lr-AdamW/pg1‚0.0004174960044138001 & lr-AdamW/pg2‚0.0004174960044138001 trainer/global_step‚3749 % _timestamp‚1.7843435367974575e+09±üQ[}2ÿ- _wandbruntime‚3195  _runtime‚3195ó»à'€ÚO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566628255223808 _step‚306 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495771648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8066076 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235668 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495771648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962352476160 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125625013643264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14638387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573518137393152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693815296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚3195.495328217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4005534791168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3956304685778 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137393152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433441 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚570584559909586 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088Y~ÒíßO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29709854 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3962352476160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8066076 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235668 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495771648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962352476160 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125625013643264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14638387 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573518137393152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693815296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚3195.495328217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4005534791168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3956304685778 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137393152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433441 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚570584559909586 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566628255223808 _step‚306 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495771648 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3957656289502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706449408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665020818432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚570732081635180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732651 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚3749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14641939 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4006559804416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843435374131854e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566774782935040 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6574056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129664911539200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450139 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450139 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚570732081635180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732651 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚3749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14641939 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4006559804416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843435374131854e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566774782935040 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6574056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129664911539200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450139 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450139 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3957656289502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706449408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665020818432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438³رo2…- _wandbruntime‚3196  _runtime‚3196+-eX†ý /loss/FBt2/detection_loss‚0.5570418262616434 _runtime‚3196.109255712 0loss/FRt2/assignment_loss‚0.9867938001390913 /loss/FRt1/detection_loss‚0.6145120436799585 %loss/total_loss‚9.197969724555513 trainer/global_step‚3749 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5963901354765339 1loss/FBt2/assignment_loss‚0.01061673124416203 1loss/SRqqt1/detection_loss‚1.1360677762388471 /loss/FBt1/detection_loss‚0.7981982050696388 _step‚308 1loss/SRqqt2/assignment_loss‚0.534419973882005 /loss/FRt2/detection_loss‚1.3697008755296034 % _timestamp‚1.7843435374159412e+09 .loss/FRt1/assignment_loss‚1.88580836708013 0loss/FBt1/assignment_loss‚0.0565869983531575 1loss/SRqqt2/detection_loss‚0.6518329916007426¤pÚM‡‚ %loss/total_loss‚9.197969724555513 trainer/global_step‚3749 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5963901354765339 1loss/FBt2/assignment_loss‚0.01061673124416203 1loss/SRqqt1/detection_loss‚1.1360677762388471 /loss/FBt1/detection_loss‚0.7981982050696388 _step‚308 1loss/SRqqt2/assignment_loss‚0.534419973882005 /loss/FRt2/detection_loss‚1.3697008755296034 % _timestamp‚1.7843435374159412e+09 .loss/FRt1/assignment_loss‚1.88580836708013 0loss/FBt1/assignment_loss‚0.0565869983531575 1loss/SRqqt2/detection_loss‚0.6518329916007426 /loss/FBt2/detection_loss‚0.5570418262616434 _runtime‚3196.109255712 0loss/FRt2/assignment_loss‚0.9867938001390913 /loss/FRt1/detection_loss‚0.6145120436799585´¡ßÖ œˆ‚(:‘ ÿ×ëÒ¢£‚” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯b”À‰‚(:µ ÿ×ëÒ ¦‘• proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.out‚ 86.26953125 disk.sda2.out‚0 disk.sda3.out‚ 86.26953125 disk.rbd1.in‚0) proc.memory.percent‚16.65205955505371 disk.dm-1.in‚0 disk.dm-1.out‚ 86.26953125" disk.nvme0c0n1.in‚ 7028.7265625' proc.memory.availableMB‚ 490.3984375" proc.memory.rssMB‚ 2728.2734375 network.recv‚868322 disk.rbd1.out‚ 8.20703125 disk.sda1.in‚0 cpu‚26.098862476306994 disk.drbd2165.out‚0! disk.dm-0.out‚8961.5732421875 network.sent‚ 5.778532e+06) disk./.usagePercent‚39.89235431041319 disk.nvme0n1.in‚ 7028.7265625$ disk.nvme0n1.out‚8961.5732421875 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚97.00684547424316$ disk./.usageGB‚594.4249076843262& disk.nvme0c0n1.out‚8961.5732421875 disk.dm-0.in‚ 7028.7265625 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0ž;Ë“Š‚(:ˆ ŽØëÒȱ‘• gpu.0.gpu‚72.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k}k®‹‚(:£ ŽØëÒÕ„ø• disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 86.4453125) proc.memory.percent‚16.65205955505371$ disk./.usageGB‚594.4248733520508 disk.rbd1.out‚ 8.20703125 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 8969.234375 disk.nvme0n1.in‚ 7140.859375" proc.memory.rssMB‚ 2728.2734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 8969.234375 disk.nvme0n1.out‚ 8969.234375 proc.cpu.threads‚17 network.recv‚872695! disk.nvme0c0n1.in‚ 7140.859375 disk.sda.in‚0$ memory_percent‚97.00531959533691 cpu‚26.09701019525191* disk./.usagePercent‚39.892352006345384 disk.dm-0.in‚ 7140.859375 disk.sda.out‚ 86.4453125 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 5.841486e+06 disk.sda1.out‚0 disk.dm-1.out‚ 86.4453125 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 490.6484375= Iw2Œ- _wandbruntime‚3235  _runtime‚3235„¼‹Éà ' lr-AdamW/pg1‚0.00041656957470066454 ' lr-AdamW/pg2‚0.00041656957470066454 trainer/global_step‚3799 % _timestamp‚1.7843435768746164e+09 _runtime‚3235.568291493 _step‚309÷ÐŒ¼ÎŽÈ _runtime‚3235.568291493 _step‚309 ' lr-AdamW/pg1‚0.00041656957470066454 ' lr-AdamW/pg2‚0.00041656957470066454 trainer/global_step‚3799 % _timestamp‚1.7843435768746164e+09µIWÐ=2- _wandbruntime‚3235  _runtime‚3235ÇzXt¸ ÚO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168060 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990094 % _timestamp‚1.7843435768757591e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242755072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8169039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14828597 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884376576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047977984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6525 kŽ(69493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30091456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013389317120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30091454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4056621446656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884376576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259514 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚3799 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35178037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242755072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131353073016320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389317120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8169307 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353182302720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚3235.569489576 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269ŸÖ+å'‘ßO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242755072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131353073016320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚310 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389317120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8169307 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353182302720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚3235.569489576 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168060 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990094 % _timestamp‚1.7843435768757591e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242755072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8169039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14828597 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884376576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047977984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30091456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013389317120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30091454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4056621446656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884376576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259514 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚3799 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35178037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592¶Æcƒí2’- _wandbruntime‚3236  _runtime‚3236ÔÎEê&“äM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4007932226408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401951232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 % _timestamp‚1.7843435774906123e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060604928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35177112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4008272776410 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8171104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187103 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401951232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305050 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35177112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚574100299489280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578108231715688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30099088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4014060604928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286191 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚577065094467584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577065094467584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4057645287424 _runtime‚3236.184437954 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138033152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8171364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6661306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14832670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14832407 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968À´>´”éM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6661306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14832670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14832407 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4007932226408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401951232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 % _timestamp‚1.7843435774906123e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060604928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35177112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4008272776410 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8171104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187103 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401951232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305050 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35177112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚574100299489280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578108231715688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30099088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4014060604928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286191 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚577065094467584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577065094467584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4057645287424 _runtime‚3236.184437954 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138033152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8171364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508·=¢é÷2•- _wandbruntime‚3236  _runtime‚3236žÑð–ÿ /loss/FBt1/detection_loss‚0.8436693269414482 epoch‚9 % _timestamp‚1.7843435774939816e+09 2loss/SRqqt1/assignment_loss‚0.5876059176916123 2loss/SRqqt2/assignment_loss‚0.4963965787744203 0loss/SRqqt1/detection_loss‚1.026612643490988 _runtime‚3236.187310698 /loss/FRt1/assignment_loss‚1.815713948143295 1loss/FBt2/assignment_loss‚0.01799101280202369 /loss/FRt1/detection_loss‚0.5774174501420317 /loss/FRt2/detection_loss‚1.2696626201794197 /loss/FBt2/detection_loss‚0.4366222702575074 %loss/total_loss‚8.723111501097302 trainer/global_step‚3799 2loss/FBt1/assignment_loss‚0.005144474306318719 _step‚312 /loss/FRt2/assignment_loss‚1.021882457999595 1loss/SRqqt2/detection_loss‚0.6243928003686425ÓÆÈ —„ /loss/FRt1/assignment_loss‚1.815713948143295 1loss/FBt2/assignment_loss‚0.01799101280202369 /loss/FRt1/detection_loss‚0.5774174501420317 /loss/FRt2/detection_loss‚1.2696626201794197 /loss/FBt2/detection_loss‚0.4366222702575074 %loss/total_loss‚8.723111501097302 trainer/global_step‚3799 2loss/FBt1/assignment_loss‚0.005144474306318719 _step‚312 /loss/FRt2/assignment_loss‚1.021882457999595 1loss/SRqqt2/detection_loss‚0.6243928003686425 /loss/FBt1/detection_loss‚0.8436693269414482 epoch‚9 % _timestamp‚1.7843435774939816e+09 2loss/SRqqt1/assignment_loss‚0.5876059176916123 2loss/SRqqt2/assignment_loss‚0.4963965787744203 0loss/SRqqt1/detection_loss‚1.026612643490988 _runtime‚3236.187310698¸Þ—ž˜‚(:“ ØëÒ¿­ ” gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0quíÈ™‚(:½ ØëÒÝ€ž– disk.drbd2165.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2728.2734375% disk.nvme0n1.out‚8980.08154296875 network.sent‚ 5.846459e+06* disk./.usagePercent‚39.892374791015925 disk.sda2.out‚0! disk.nvme0n1.in‚ 7141.69140625 disk.rbd1.in‚0 disk.rbd1.out‚ 8.33203125$ memory_percent‚97.00706005096436) proc.memory.percent‚16.65205955505371 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 7141.69140625' disk.nvme0c0n1.out‚8980.08154296875 disk.dm-0.in‚ 7141.69140625 cpu‚26.095259880395343 disk.sda.in‚0 disk.sda.out‚ 86.58203125 disk.sda3.out‚ 86.58203125 network.recv‚876306 disk.drbd2165.in‚0" disk.dm-0.out‚8980.08154296875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 86.58203125( proc.memory.availableMB‚ 490.36328125$ disk./.usageGB‚594.4252128601074²Ã"w‘š‚(:† ¬ØëÒ™ðˆ” gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šîËæÀ›‚(:µ ¬ØëÒÊð¿• disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 86.9296875! disk.dm-0.out‚8982.9033203125" proc.memory.rssMB‚ 2728.2734375 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7141.69140625 disk.dm-0.in‚ 7141.69140625 network.sent‚ 5.909689e+06 disk.sda2.in‚0 disk.sda3.out‚ 86.9296875 disk.rbd1.in‚0) proc.memory.percent‚16.65205955505371 proc.cpu.threads‚17 network.recv‚880841$ disk./.usageGB‚594.4251365661621 disk.rbd0.out‚0! disk.nvme0n1.in‚ 7141.69140625$ disk.nvme0n1.out‚8982.9033203125 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.093542704093885 disk.sda.out‚ 86.9296875& disk.nvme0c0n1.out‚8982.9033203125$ memory_percent‚97.00665473937988' proc.memory.availableMB‚ 490.4296875) disk./.usagePercent‚39.89236967086525 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 8.33203125òƒ£àœ‚(:’ »ØëÒ¥ëГ gpu.0.gpu‚78.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕòÓϺ‚(:¯ »ØëÒÑ¡¡•) disk./.usagePercent‚39.89237223094059 disk.sda3.out‚ 87.1484375) proc.memory.percent‚16.65205955505371( proc.memory.availableMB‚ 490.43359375 disk.sda.in‚0$ disk./.usageGB‚594.4251747131348 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 7142.65625 cpu‚26.091757168446254 network.recv‚884357" disk.dm-0.out‚8992.21044921875 disk.dm-1.in‚0 disk.dm-1.out‚ 87.1484375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚8992.21044921875 disk.dm-0.in‚ 7142.65625 disk.sda.out‚ 87.1484375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 7142.65625% disk.nvme0n1.out‚8992.21044921875 disk.rbd1.out‚ 8.39453125$ memory_percent‚97.00663089752197 network.sent‚ 5.91479e+06 disk.sda1.out‚0" proc.memory.rssMB‚ 2728.2734375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17˜e¢\2ž- _wandbruntime‚3275  _runtime‚3275ùö »ÇŸÁ _runtime‚3275.477558375 _step‚313 & lr-AdamW/pg1‚0.0004156308111885987 & lr-AdamW/pg2‚0.0004156308111885987 trainer/global_step‚3849 % _timestamp‚1.7843436167839806e+09³­ÝCÌ Æ & lr-AdamW/pg1‚0.0004156308111885987 & lr-AdamW/pg2‚0.0004156308111885987 trainer/global_step‚3849 % _timestamp‚1.7843436167839806e+09 _runtime‚3275.477558375 _step‚313¹»«ž•2¡- _wandbruntime‚3275  _runtime‚3275DÏÂÀà'¢ÚO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15019504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4063743479808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588343989738496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35622512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚314 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4107790087680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084818944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095566 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35632489 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8272436 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584279887880192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6746811 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659852 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585336860070602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843436167850742e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133075679583744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133075782578688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631360000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743479808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3275.478800972 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚3849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588343989738496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35622512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚314 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4107790087680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084818944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095566 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35632489 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8272436 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584279887880192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6746811 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659852 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585336860070602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843436167850742e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133075679583744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133075782578688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631360000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743479808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3275.478800972 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚3849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ºW.•|2¤- _wandbruntime‚3276  _runtime‚3276·'" ¥äM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚315 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112250 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065097453056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122243 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceSÏ}=à!tatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585484381796196 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584427128909824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584427128909824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35631564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133110122547200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15023063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243402240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4064756106752 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756106752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8274510 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641555 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15023335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110231826432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30480688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108894959616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097453056 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269ûž#ï&¦éM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122243 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585484381796196 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584427128909824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584427128909824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35631564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133110122547200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15023063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243402240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4064756106752 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756106752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8274510 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641555 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15023335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110231826432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30480688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108894959616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097453056 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚315 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112250 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065097453056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8375982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4158929240576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4108154146502 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086941 JDDeviceStatsMonito÷ÄÜ] r.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378343424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378343424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780320768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376½Ý÷bå'µßO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134798275275264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843436569230747e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134798163891712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8375982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4158929240576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4108154146502 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086941 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378343424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378343424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780320768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚591641939361792 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4115451608576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134832605800448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595904632000000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958295 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588751366676480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134832719273984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15213593 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚3316.232501761 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792954880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792954880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096007 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4115451608576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15213869 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595904990385664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4115451608576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134832605800448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595904632000000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958295 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588751366676480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134832719273984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15213593 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚3316.232501761 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792954880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792954880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096007 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4115451608576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15213869 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595904990385664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784343697102016e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483705344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚603169483705344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820868 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3949 _runtime‚ 3355.79573739 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125326848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4165134483456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8479343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 _step‚322 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31236254 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36531416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136520807234048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600089160231618 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599003990843392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144oÌæsã'ÅÝO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784343697102016e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483705344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚603169483705344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820868 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3949 _runtime‚ 3355.79573739 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125326848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4165134483456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8479343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 _step‚322 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31236254 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36531416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136520807234048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600089160231618 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599003990843392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4209924210176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595930389598208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15400654 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8479066 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595930406637568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376´Þ.œ2Æ- _wandbruntime‚3356  _runtime‚3356’S)2OÇäM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136555140560896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚603317737369088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132344212719616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4166147110400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚323 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15404162 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8481109 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4210946453504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4211037120512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147110400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚Ù{(ë› -1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603317737369088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546574 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36540468 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599151248912384/3éï&ÈéM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599151248912384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136555140560896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚603317737369088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132344212719616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4166147110400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚323 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15404162 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8481109 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4210946453504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4211037120512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147110400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603317737369088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546574 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36540468 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392ÃQr£'2É- _wandbruntime‚3356  _runtime‚3356ˆC„ ʃ 2loss/FBt1/assignment_loss‚0.020273634388559003 /loss/FRt1/detection_loss‚0.5739562131186021 /loss/FBt2/detection_loss‚0.5968421943446932 _runtime‚3356.412682135 /loss/FRt1/assignment_loss‚1.787630110274606 2loss/SRqqt1/assignment_loss‚0.5250352874275572 %loss/total_loss‚8.718789249289872 epoch‚9 trainer/global_step‚3949 % _timestamp‚1.7843436977194026e+09 _step‚324 3loss/SRqqt2/assignment_loss‚0.45424998285402096 2loss/FBt2/assignment_loss‚0.026465540949336248 /loss/FRt2/detection_loss‚1.3338995138745562 1loss/SRqqt1/detection_loss‚0.9522333984776544 1loss/SRqqt2/detection_loss‚0.6188283897621283 0loss/FRt2/assignment_loss‚0.9390786067692344 /loss/FBt1/detection_loss‚0.8902963770489241f·Éðˈ /loss/FRt1/assignment_loss‚1.787630110274606 2loss/SRqqt1/assignment_loss‚0.5250352874275572 %loss/total_loss‚8.718789249289872 epoch‚9 trainer/global_step‚3949 % _timestamp‚1.7843436977194026e+09 _step‚324 3loss/SRqqt2/assignment_loss‚0.45424998285402096 2loss/FBt2/assignment_loss‚0.026465540949336248 /loss/FRt2/detection_loss‚1.3338995138745562 1loss/SRqqt1/detection_loss‚0.9522333984776544 1loss/SRqqt2/detection_loss‚0.6188283897621283 0loss/FRt2/assignment_loss‚0.9390786067692344 /loss/FBt1/detection_loss‚0.8902963770489241 2loss/FBt1/assignment_loss‚0.020273634388559003 /loss/FRt1/detection_loss‚0.5739562131186021 /loss/FBt2/detection_loss‚0.5968421943446932 _runtime‚3356.412682135ÄÙ~<@“Ì‚(:ˆ •ÙëÒ¥½Ù“ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø§ýf¾Í‚(:³ •ÙëÒ¬æ˜ disk.drbd2165.out‚0 disk.nvme0n1.in‚ 7341.1328125 disk.rbd0.in‚0) proc.memory.percent‚16.08710289001465" disk.nvme0c0n1.in‚ 7341.1328125" disk.dm-0.out‚9038.91455078125 network.sent‚ 6.060393e+06! disk./.usageGB‚594.4248046875 disk.drbd2165.in‚0 disk.dm-0.in‚ 7341.1328125 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 cpu‚26.081669175065578 disk.dm-1.in‚0 disk.sda.out‚ 89.44921875 disk.sda2.in‚0 disk.sda3.out‚ 89.44921875 disk.rbd1.out‚ 8.58203125 disk.rbd0.out‚0 network.recv‚907369" proc.memory.rssMB‚ 2635.7109375' disk.nvme0c0n1.out‚9038.91455078125 disk.sda3.in‚0$ memory_percent‚96.49162292480469$ proc.memory.availableMB‚574.8125* disk./.usagePercent‚39.892347398209765 disk.dm-1.out‚ 89.44921875% disk.nvme0n1.out‚9038.91455078125è ŽØ’Î‚(:‡ ¤ÙëÒö–é• gpu.0.gpu‚24.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y®Ö½Ï‚(:² ¤ÙëÒ‚“˜ disk.sda1.in‚0 disk.dm-1.out‚89.96875' disk.nvme0c0n1.out‚9046.20654296875' proc.memory.availableMB‚ 574.8671875 network.sent‚ 6.125266e+06 network.recv‚911451 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0) proc.memory.percent‚16.08710289001465$ disk./.usageGB‚594.4247817993164 disk.rbd0.out‚0 disk.sda3.out‚89.96875 cpu‚26.079993180604415 disk.sda2.out‚0! disk.nvme0n1.in‚ 7341.96484375# disk.nvme0c0n1.in‚ 7341.96484375" disk.dm-0.out‚9046.20654296875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2635.7109375 disk.rbd0.in‚0% disk.nvme0n1.out‚9046.20654296875 disk.rbd1.out‚ 8.58203125$ memory_percent‚96.49128913879395 proc.cpu.threads‚17) disk./.usagePercent‚39.89234586216456 disk.dm-0.in‚ 7341.96484375 disk.sda.out‚89.96875©.µrЂ(:’ ³ÙëÒ«„ß” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò-}®ÇÑ‚(:¼ ³ÙëÒÒØ’– disk.dm-0.in‚ 7341.96484375" disk.dm-0.out‚9051.35693359375 disk.sda2.in‚0! disk.nvme0n1.in‚ 7341.96484375 disk.sda2.out‚0 network.sent‚ 6.128346e+06 proc.cpu.threads‚17 disk.sda3.out‚ 90.19140625 disk.sda1.out‚0 disk.sda.in‚0) proc.memory.percent‚16.08710289001465# disk.nvme0c0n1.in‚ 7341.96484375 cpu‚26.078284676408554 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 90.19140625$ disk./.usageGB‚594.4246025085449 disk.rbd1.in‚0 disk.rbd1.out‚ 8.70703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚9051.35693359375" proc.memory.rssMB‚ 2635.7109375 network.recv‚914859 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 90.19140625% disk.nvme0n1.out‚9051.35693359375$ memory_percent‚96.49279117584229( proc.memory.availableMB‚ 574.62109375) disk./.usagePercent‚39.89233382981045÷ê|ù2Ò- _wandbruntime‚3395  _runtime‚3395°Ûÿ–ÉÓà % _timestamp‚1.7843437370279484e+09 _runtime‚3395.721549004 _step‚325 ' lr-AdamW/pg1‚0.00041274112474667956 ' lr-AdamW/pg2‚0.00041274112474667956 trainer/global_step‚3999«ýÓÏÎÔÈ ' lr-AdamW/pg1‚0.00041274112474667956 ' lr-AdamW/pg2‚0.00041274112474667956 trainer/global_step‚3999 % _timestamp‚1.7843437370279484e+09 _runtime‚3395.721549004 _step‚325Å +ã2Õ- _wandbruntime‚3395  _runtime‚3395·šgVà'ÖÚO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312126 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829985280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4261171579392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4261087196672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985868 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603255923191808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31617856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872310272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603722 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31617854 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613701 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995845 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985868 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚326 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15591708 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8582891 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843437370290678e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138243393466880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872310272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209387120318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15591449 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138243290471936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829985280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171324416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7008561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606366042324992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0À¹–h××ßO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843437370290678e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138243393466880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872310272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209387120318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch¯ _start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15591449 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138243290471936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829985280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171324416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7008561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606366042324992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312126 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829985280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4261171579392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4261087196672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985868 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603255923191808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31617856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872310272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603722 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31617854 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613701 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995845 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985868 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚326 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15591708 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8582891 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ÆþÃú«2Ø- _wandbruntime‚3396  _runtime‚3396E³$ê&ÙäM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8585136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603402450903040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4262114916352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610730484352512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610730125966848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31625486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚606513283354624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15595184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4217183958528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603402433863680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37004911 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15595442 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607612832037720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843437376474261e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚606513300393984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚606513283354624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36994920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138277746950144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4262199291904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31625488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607613189627082 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7010306 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216842612224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8584881 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3999 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210398174040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31625486 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842612224…ï&ÚéM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610730125966848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31625486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚606513283354624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15595184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4217183958528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603402433863680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37004911 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15595442 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607612832037720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843437376474261e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚606513300393984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚606513283354624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36994920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138277746950144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4262199291904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31625488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607613189627082 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7010306 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216842612224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8584881 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3999 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210398174040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31625486 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8585136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603402450903040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4262114916352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610730484352512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5740628838539124 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285336494445801 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7485453486442566 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 +"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937450885772705 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961322546005249 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7268894910812378 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03192221000790596 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6903164386749268 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 +"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6644415259361267 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Î¥˜«ý0Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8285745978355408 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2910749018192291 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 ,jet/accuracy_4_of_4‚0.013899499550461769 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9839286804199219 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6481934785842896 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8359032869338989 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46486395597457886 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21227623522281647 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996992290019989 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406060457229614 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951722025871277 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10097356885671616 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5805191397666931 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8326342701911926 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8645107746124268 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458333492279053 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028175251558423042 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880961179733276 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7460162043571472 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7336801886558533 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_3_of_4‚0.06134622544050217 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4868696331977844 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22334259748458862 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7625865936279297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.703097403049469 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6926009654998779 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2771734595298767 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9883680939674377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03215419873595238 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7028811573982239 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10093952715396881 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7635239362716675 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6997926235198975 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_2_of_3‚0.532902181148529 (particle/accuracy‚0.8443236351013184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8099743127822876 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6066915392875671 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041969265788793564 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19824473559856415 +particle/specificity‚0.9093109965324402 6validation_average_jet_accuracy‚0.3831193149089813 epoch‚9 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14511516690254211 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23314730823040009 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04977129027247429 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6275151968002319 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3497389256954193 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928193688392639 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6527683138847351 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_3_of_3‚0.06094259396195412 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_2_of_2‚0.9998962879180908 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 )jet/accuracy_2_of_4‚0.764133095741272 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6346182227134705 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6395541429519653 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2621403634548187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7644216418266296 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.655351459980011 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6551135778427124 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8319539427757263 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2311987578868866 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03386850282549858 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.684449315071106 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7394277453422546 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6065555810928345 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.025860212743282318 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843437548420708e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3420175611972809 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961322546005249 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7337679862976074 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800323963165283 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5020706653594971 particle/accuracy_1_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38449960947036743 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964068531990051 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8065570592880249 _step‚329 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829018712043762 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23233363032341003 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11968308687210083 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47831788659095764 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.048854760825634 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8685657978057861 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20893166959285736 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2128640115261078 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800121784210205 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27009302377700806 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2190484255552292 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7582603096961975 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8125222325325012 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/f_score‚0.6259900331497192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992874801158905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6779179573059082 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 trainer/global_step‚3999 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9348187446594238 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 /particle/accuracy_4_of_4‚0.9225271344184875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6930862069129944 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940983057022095 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028175251558423042 *jet/accuracy_1_of_4‚0.9838173389434814 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28882184624671936 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031115027144551277 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08252078294754028 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03384488448500633 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6527683138847351 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23535698652267456 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28763583302497864 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21516616642475128 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34634310007095337 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6586686968803406 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7737449407577515 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3867250680923462 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8645107746124268 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3154919147491455 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3875146508216858 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0` Ê=âÄ{ 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2771734595298767 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9883680939674377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03215419873595238 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7028811573982239 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10093952715396881 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7635239362716675 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6997926235198975 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_2_of_3‚0.532902181148529 (particle/accuracy‚0.8443236351013184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8099743127822876 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6066915392875671 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041969265788793564 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19824473559856415 +particle/specificity‚0.9093109965324402 6validation_average_jet_accuracy‚0.3831193149089813 epoch‚9 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14511516690254211 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23314730823040009 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04977129027247429 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6275151968002319 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3497389256954193 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928193688392639 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6527683138847351 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_3_of_3‚0.06094259396195412 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_2_of_2‚0.9998962879180908 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 )jet/accuracy_2_of_4‚0.764133095741272 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6346182227134705 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6395541429519653 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2621403634548187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7644216418266296 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.655351459980011 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6551135778427124 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8319539427757263 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2311987578868866 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03386850282549858 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.684449315071106 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7394277453422546 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6065555810928345 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.025860212743282318 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843437548420708e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3420175611972809 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961322546005249 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7337679862976074 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800323963165283 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5020706653594971 particle/accuracy_1_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38449960947036743 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964068531990051 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8065570592880249 _step‚329 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829018712043762 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23233363032341003 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11968308687210083 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47831788659095764 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.048854760825634 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8685657978057861 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20893166959285736 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2128640115261078 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800121784210205 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27009302377700806 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2190484255552292 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7582603096961975 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8125222325325012 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/f_score‚0.6259900331497192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992874801158905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6779179573059082 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 trainer/global_step‚3999 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9348187446594238 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 /particle/accuracy_4_of_4‚0.9225271344184875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6930862069129944 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940983057022095 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028175251558423042 *jet/accuracy_1_of_4‚0.9838173389434814 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28882184624671936 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031115027144551277 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08252078294754028 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03384488448500633 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6527683138847351 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23535698652267456 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28763583302497864 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21516616642475128 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34634310007095337 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6586686968803406 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7737449407577515 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3867250680923462 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8645107746124268 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3154919147491455 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3875146508216858 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9175232648849487 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2206301987171173 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6727076768875122 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3321521580219269 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.650323748588562 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4294193387031555 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2434762567281723 jet/accuracy_1_of_5‚1 (loss/val_total_loss‚8.52510371580623 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8735039234161377 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2800323963165283 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7425456047058105 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2131073772907257 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7442977428436279 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06418067961931229 particle/accuracy_1_of_5‚1 _runtime‚3413.536283972 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5740628838539124 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8285336494445801 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7485453486442566 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 +"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937450885772705 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961322546005249 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7268894910812378 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03192221000790596 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6903164386749268 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 +"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6644415259361267 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8285745978355408 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2910749018192291 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 ,jet/accuracy_4_of_4‚0.013899499550461769 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9839286804199219 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6481934785842896 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8359032869338989 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46486395597457886 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21227623522281647 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996992290019989 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406060457229614 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951722025871277 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10097356885671616 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5805191397666931 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8326342701911926 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8645107746124268 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458333492279053 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028175251558423042 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9880961179733276 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7460162043571472 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7336801886558533 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_3_of_4‚0.06134622544050217 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4868696331977844 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22334259748458862 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7625865936279297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.703097403049469 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6926009654998779 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925ÉGnUû“ã‚(:ˆ ÑÙëÒù¦¢• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:æ~ÁÄä‚(:¹ ÑÙëÒÚñ¾–% disk.nvme0n1.out‚9314.50146484375# proc.memory.rssMB‚ 2716.51171875$ disk./.usageGB‚594.4256172180176 disk.sda1.in‚0 network.sent‚ 6.196087e+06 disk.rbd0.out‚0 disk.sda2.out‚0$ proc.memory.availableMB‚743.5625 proc.cpu.threads‚17 network.recv‚922429) disk./.usagePercent‚39.89240192781456# disk.nvme0c0n1.in‚ 7436.92578125 disk.dm-0.in‚ 7436.92578125 disk.sda.out‚ 90.71484375! disk.nvme0n1.in‚ 7436.92578125 cpu‚26.093763554658445' disk.nvme0c0n1.out‚9314.50146484375 disk.sda3.out‚ 90.71484375 disk.rbd1.in‚0 disk.rbd1.out‚ 8.76953125" disk.dm-0.out‚9314.50146484375$ memory_percent‚95.46165466308594 disk.sda.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.58027172088623 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 90.71484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0"°#žå‚(:“ àÙëÒ¸¯Ð“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@‘3¾æ‚(:³ àÙëÒÄ‹›— disk.rbd0.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚9562.2646484375 disk.sda3.out‚ 90.953125 proc.cpu.threads‚17$ disk./.usageGB‚594.4246635437012 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0! disk.dm-0.out‚9562.2646484375 disk.sda3.in‚0( proc.memory.availableMB‚ 743.37890625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 7436.92578125 cpu‚26.09204698457438 disk.rbd0.out‚0 disk.sda2.in‚0$ memory_percent‚95.46277523040771 network.recv‚926615 disk.rbd1.out‚ 8.76953125# proc.memory.rssMB‚ 2716.51171875 network.sent‚ 6.25334e+06* disk./.usagePercent‚39.892337925930995 disk.dm-1.out‚ 90.953125 disk.sda.in‚0! disk.nvme0n1.in‚ 7436.92578125$ disk.nvme0n1.out‚9562.2646484375) proc.memory.percent‚16.58027172088623 disk.dm-0.in‚ 7436.92578125 disk.sda.out‚ 90.953125 disk.sda2.out‚0®Ë˜‡’ç‚(:‡ ïÙëÒí©Ê“ gpu.0.gpu‚25.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ã±»è‚(:° ïÙëÒÊͽ– network.recv‚930280 disk.drbd2165.out‚0 disk.rbd1.out‚ 8.83203125 disk.sda2.out‚0 disk.drbd2165.in‚0! disk.dm-0.out‚9571.9912109375 disk.sda2.in‚0 disk.sda3.out‚ 91.1796875' proc.memory.availableMB‚ 743.3984375# proc.memory.rssMB‚ 2716.51171875 proc.cpu.threads‚17 network.sent‚ 6.258483e+06& disk.nvme0c0n1.out‚9571.9912109375 disk.dm-0.in‚ 7437.890625$ disk.nvme0n1.out‚9571.9912109375 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚26.090440126437038 disk.sda.out‚ 91.1796875 disk.nvme0n1.in‚ 7437.890625 disk.sda3.in‚0$ disk./.usageGB‚594.4242897033691! disk.nvme0c0n1.in‚ 7437.890625 disk.sda1.out‚0 disk.dm-1.out‚ 91.1796875 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚95.46265602111816) proc.memory.percent‚16.58027172088623) disk./.usagePercent‚39.89231283719265 disk.sda.in‚0 disk.rbd0.out‚0_ø››2é- _wandbruntime‚3455  _runtime‚3455Oà÷¾Áêà ' lr-AdamW/pg1‚0.00041175363685253013 ' lr-AdamW/pg2‚0.00041175363685253013 trainer/global_step‚4049 % _timestamp‚1.7843437965454733e+09 _runtime‚3455.239046534 _stŽ´Gep‚330G\¤ÎëÈ ' lr-AdamW/pg1‚0.00041175363685253013 ' lr-AdamW/pg2‚0.00041175363685253013 trainer/global_step‚4049 % _timestamp‚1.7843437965454733e+09 _runtime‚3455.239046534 _step‚330ÊÃa0D2ì- _wandbruntime‚3455  _runtime‚3455‘utwß'íÙO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615657677496320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8718621 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8718898 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330657001984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619942277530624 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277530624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919152128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140000228915712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135669553301504 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37510647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15817229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32120120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919152128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4284600034304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4284600034304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921198 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37510647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15816949 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258695168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640746 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37520624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332460 % _timestamp‚1.7843437965467207e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4330578910720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15816949 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258695168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640746 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37520624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332460 % _timestamp‚1.7843437965467207e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4330578910720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8718898 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330657001984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619942277530624 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277530624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919152128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140000228915712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135669553301504 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37510647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15817229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32120120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919152128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4284600034304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4284600034304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921198 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Ë·©2ï- _wandbruntime‚3455  _runtime‚3455È0 íé&ðãM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7100076 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616915365057542 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647451 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8720963 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8720708 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657444 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271322112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140034689216512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531194368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32127752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331688544256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616915722646904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657444 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚620090172808704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4278734386182 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140034582034432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15820781 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 3455.85527143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647451 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612668416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843437971614912e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37529690 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135702982060032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚620090531194368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519699 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519699 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657444 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271322112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140034689216512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531194368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32127752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331688544256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616915722646904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657444 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚620090172808704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4278734386182 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140034582034432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15820781 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 3455.85527143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647451 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612668416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843437971614912e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37529690 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135702982060032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚620090531194368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PZè2ò- _wandbruntime‚3455  _runtime‚3455Yƒˆ_óÿ 1loss/FBt1/assignment_loss‚0.10956995382167828 1loss/FBt2/assignment_loss‚0.11666616497105409 1loss/SRqqt2/detection_loss‚0.6149520740206251 .loss/FBt2/detection_loss‚0.563491826905686 trainer/global_step‚4049 1loss/SRqqt1/detection_loss‚0.9030135420650831 _step‚333 .loss/FRt1/detection_loss‚0.585849198743583 /loss/FRt2/detection_loss‚1.3338922214613442 epoch‚10 2loss/SRqqt1/assignment_loss‚0.6648573237590683 2loss/SRqqt2/assignment_loss‚0.5828316527220428 .loss/FBt1/detection_loss‚1.057311383386453 %loss/total_loss‚9.405712930541416 % _timestamp‚1.7843437971636512e+09 _runtime‚3455.856941583 0loss/FRt1/assignment_loss‚1.9095926764549591 0loss/FRt2/assignment_loss‚0.9636849122298391÷úÛ ô„ 1loss/SRqqt1/detection_loss‚0.9030135420650831 _step‚333 .loss/FRt1/detection_loss‚0.585849198743583 /loss/FRt2/detection_loss‚1.3338922214613442 epoch‚10 2loss/SRqqt1/assignment_loss‚0.6648573237590683 2loss/SRqqt2/assignment_loss‚0.5828316527220428 .loss/FBt1/detection_loss‚1.057311383386453 %loss/total_loss‚9.405712930541416 % _timestamp‚1.7843437971636512e+09 _runtime‚3455.856941583 0loss/FRt1/assignment_loss‚1.9095926764549591 0loss/FRt2/assignment_loss‚0.9636849122298391 1loss/FBt1/assignment_loss‚0.10956995382167828 1loss/FBt2/assignment_loss‚0.11666616497105409 1loss/SRqqt2/detection_loss‚0.6149520740206251 .loss/FBt2/detection_loss‚0.563491826905686 trainer/global_step‚4049ÍŸügâ’õ‚(:‡ þÙëÒהؔ gpu.0.gpu‚14.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™°¦¾ö‚(:³ þÙëÒóô„–) disk./.usagePercent‚39.89231667730566 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 743.50390625 disk.sda.in‚0 disk.sda.out‚ 91.30078125 network.recv‚933545! disk.nvme0n1.in‚ 7555.36328125 disk.dm-0.in‚ 7555.5625 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.percent‚16.58027172088623 network.sent‚ 6.263467e+06 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.088841903461518 proc.cpu.threads‚17$ disk./.usageGB‚594.4243469238281# disk.nvme0n1.out‚9579.466796875 disk.sda3.out‚ 91.30078125 disk.dm-1.out‚ 91.30078125 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚9579.466796875 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 7555.36328125 disk.dm-0.out‚9579.466796875 disk.sda2.in‚0# proc.memory.rssMB‚ 2716.51171875 disk.rbd1.out‚ 8.83203125 disk.rbd0.out‚0$ memory_percent‚95.46201229095459þ±¦¹ž÷‚(:“ ÚëÒÎøÊ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"VÙÂø‚(:· ÚëÒÈÔç• disk.dm-0.in‚ 7574.16015625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 91.5546875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7574.16015625& disk.nvme0c0n1.out‚9584.5205078125 disk.sda2.out‚0! disk.nvme0n1.in‚ 7574.16015625( proc.memory.availableMB‚ 496.81640625) disk./.usagePercent‚39.89226368374607! disk.dm-0.out‚9584.5205078125 disk.sda1.in‚0 disk.rbd1.out‚ 8.89453125 network.recv‚938534 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2716.51171875 disk.sda.out‚ 91.5546875$ disk.nvme0n1.out‚9584.5205078125 disk.sda1.out‚0) proc.memory.percent‚16.58027172088623 cpu‚26.087111966581944 proc.cpu.threads‚17 network.sent‚ 6.326504e+06 disk.sda3.out‚ 91.5546875$ memory_percent‚96.96767330169678$ disk./.usageGB‚594.4235572814941 disk.sda3.in‚0 disk.drbd2165.in‚0ÖÇ/gù‚(:’ œÚëÒ‘ùˆ• gpu.0.gpu‚13.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]lZ2²ú‚(:§ œÚëÒŸÁµ– disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚26.085596472745067 disk.dm-1.out‚ 91.8359375' disk.nvme0c0n1.out‚9591.63720703125$ proc.memory.availableMB‚497.0625 disk.sda2.out‚0 network.sent‚ 6.330671e+06% disk.nvme0n1.out‚9591.63720703125 disk.sda3.in‚0 disk.nvme0c0n1.in‚7575.125$ memory_percent‚96.96617126464844# proc.memory.rssMB‚ 2716.51171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.percent‚16.58027172088623 network.recv‚941212 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚9591.63720703125 disk.nvme0n1.in‚7575.125 disk.sda3.out‚ 91.8359375 disk.rbd1.out‚ 8.89453125 disk.drbd2165.out‚0 disk.dm-0.in‚7575.125 disk.sda.out‚ 91.8359375 proc.cpu.threads‚17) disk./.usagePercent‚39.89226701184401$ disk./.usageGB‚594.4236068725586„>[]2û- _wandbruntime‚3495  _runtime‚3495 _¢ÇüÁ _runtime‚3495.380736891 _step‚334 & lr-AdamW/pg1‚0.0004107541244781391 & lr-AdamW/pg2‚0.0004107541244781391 trainer/global_step‚4099 % _timestamp‚1.7843438366871822e+09Ã/j°ÌýÆ & lr-AdamW/pg1‚0.0004107541244781391 & lr-AdamW/pg2‚0.0004107541244781391 trainer/global_step‚4099 % _timestamp‚1.7843438366871822e+09 _runtime‚3495.380736891 _step‚334Χð‚2þ- _wandbruntime‚3495  _runtime‚3495†½Tà'ÿÚO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8823614 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954196992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141722875256832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16008935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024514048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37975076 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37975076 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32501722 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666135552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335295536128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 v¯m €ßO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16009195 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7185578 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32501722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412456 trainer/global_step‚4099 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37965099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965099 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137340991227904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4381966314496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627355024514048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4335295536128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7185581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619815653593088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954196992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141722875256832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16008935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024514048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37975076 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37975076 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32501722 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 Bõ2xcated‚619815670632448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623019728977920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954196992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843438366882517e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666135552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335295536128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8823614 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4329691423092 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4336308170240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4335966823936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16012988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619962181304320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623166970007552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627502919792128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974151 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141757430008320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873090 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984142 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3496.003379001 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137374419986432 trainer/global_step‚4099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278177792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7187326 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843438373094096e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚623166952968192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727412 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70483503 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138050 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619962164264960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8825396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141757322826240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 âÏÚï&ƒéM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727412 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70483503 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138050 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619962164264960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8825396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141757322826240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4329691423092 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4336308170240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4335966823936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16012988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619962181304320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623166970007552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627502919792128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974151 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141757430008320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873090 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984142 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3496.003379001 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137374419986432 trainer/global_step‚4099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278177792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7187326 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843438373094096e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚623166952968192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0а¼F2„- _wandbruntime‚3496  _runtime‚3496èB…ÿ 2loss/SRqqt1/assignment_loss‚0.5825422188475051 3loss/FBt1/assignment_loss‚0.0013507289620221324 1loss/SRqqt2/detection_loss‚0.6946203635176927 .loss/FBt2/detection_loss‚0.582986256656739 $loss/total_loss‚8.86909447659276 0loss/FRt1/assignment_loss‚1.7743883084216343 .loss/FRt1/detection_loss‚0.588833004216284 1loss/SRqqt1/detection_loss‚0.9104218305376555 /loss/FBt1/detection_loss‚0.9367057600551668 0loss/FRt2/assignment_loss‚0.9305930528716136 2loss/SRqqt2/assignment_loss‚0.4693678883226436 /loss/FRt2/detection_loss‚1.3486409145407954 # _timestamp‚1.78434383731067e+09 _step‚337 2loss/FBt2/assignment_loss‚0.048644149643007915 epoch‚10 trainer/global_step‚4099 _runtime‚ 3496.00400893hæ! †„ 3loss/FBt1/assignment_loss‚0.0013507289620221324 1loss/SRqqt2/detection_loss‚0.6946203635176927 .loss/FBt2/detection_loss‚0.582986256656739 $loss/total_loss‚8.86909447659276 0loss/FRt1/assignment_loss‚1.7743883084216343 .loss/FRt1/detection_loss‚0.588833004216284 1loss/SRqqt1/detection_loss‚0.9104218305376555 /loss/FBt1/detection_loss‚0.9367057600551668 0loss/FRt2/assignment_loss‚0.9305930528716136 2loss/SRqqt2/assignment_loss‚0.4693678883226436 /loss/FRt2/detection_loss‚1.3486409145407954 # _timestamp‚1.78434383731067e+09 _step‚337 2loss/FBt2/assignment_loss‚0.048644149643007915 epoch‚10 trainer/global_step‚4099 _runtime‚ 3496.00400893 2loss/SRqqt1/assignment_loss‚0.5825422188475051Ñ‹{…›‡‚(: «ÚëҮ̓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏD,Ɉ‚(:· «ÚëÒâÌ×— network.recv‚944814$ disk./.usageGB‚594.4256057739258 disk.dm-0.in‚ 7575.51953125 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚26.084013377029773! disk.nvme0n1.in‚ 7575.51953125% disk.nvme0n1.out‚9595.70556640625 disk.rbd1.in‚0 disk.sda.out‚ 92.140625 disk.sda3.out‚ 92.140625 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.58027172088623 proc.cpu.threads‚17 network.sent‚ 6.348551e+06) disk./.usagePercent‚39.89240115979196 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7575.51953125' disk.nvme0c0n1.out‚9595.70556640625" disk.dm-0.out‚9595.70556640625 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 92.140625$ memory_percent‚96.96495532989502( proc.memory.availableMB‚ 497.26171875# proc.memory.rssMB‚ 2716.51171875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 8.95703125 disk.rbd0.out‚0ûѬž‰‚(:“ ºÚëÒô‡¥• gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wUN¸Š‚(:­ ºÚëÒ®©þ–# disk.nvme0c0n1.out‚ 9609.0234375 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚949245' proc.memory.availableMB‚ 497.2578125 disk.sda1.in‚0 disk.dm-1.out‚ 92.4453125 disk.dm-0.in‚ 7575.95703125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 6.398388e+06 disk.dm-0.out‚ 9609.0234375! disk.nvme0n1.in‚ 7575.95703125# proc.memory.rssMB‚ 2716.51171875) proc.memory.percent‚16.58027172088623$ disk./.usageGB‚594.4291381835938# disk.nvme0c0n1.in‚ 7575.95703125! disk.nvme0n1.out‚ 9609.0234375 disk.sda3.out‚ 92.4453125 cpu‚26.082332817955525) disk./.usagePercent‚39.89263822276866 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚96.96497917175293 disk.sda.out‚ 92.4453125 disk.rbd1.out‚ 8.95703125ôî¢æ2‹- _wandbruntime‚3535  _runtime‚3535„~ÁÆŒÀ & lr-AdamW/pg1‚0.0004097426518203187 & lr-AdamW/pg2‚0.0004097426518203187 trainer/global_step‚4149 $ _timestamp‚1.784343876839389e+09 _runtime‚3535.534369766 _step‚338 *uÒËÅ _runtime‚3535.534369766 _step‚338 & lr-AdamW/pg1‚0.0004097426518203187 & lr-AdamW/pg2‚0.0004097426518203187 trainer/global_step‚4149 $ _timestamp‚1.784343876839389e+09Òhw 92Ž- _wandbruntime‚3535  _runtime‚3535ø ñd$ÚO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16200171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143445502965248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚339 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419551 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8927595 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843438768401854e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4385649698816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771497472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312850 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16200426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306276 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991037952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pä-m¼ool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413118976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991037952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432964524544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413118976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143445393678848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419551 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649698816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8927343 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413118976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143445393678848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419551 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649698816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8927343 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚339 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419551 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8927595 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843438768401854e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4385649698816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771497472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312850 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16200426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306276 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991037952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413118976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991037952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432964524544Ó€›,2‘- _wandbruntime‚3536  _runtime‚3536K58ê&’äM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319555 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32890952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433981487616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚340 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025161216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38428603 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967359998 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚3536.150107338 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38428603 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38438594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4434076348928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16203955 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8929382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8929663 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634915666775552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4387003672064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4386662325760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843438774562733e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4387003672064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662325760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7274576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7274573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627287697858560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438594 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101yô: “éM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4387003672064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662325760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7274576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7274573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627287697858560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438594 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319555 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32890952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433981487616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚340 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025161216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38428603 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967359998 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚3536.150107338 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38428603 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38438594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4434076348928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16203955 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8929382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8929663 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634915666775552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4387003672064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4386662325760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843438774562733e+09Ô‚0ó2”- _wandbruntime‚3536  _runtime‚3536Ir“•‚ 2loss/SRqqt1/assignment_loss‚0.5721552571080617 2loss/SRqqt2/assignment_loss‚0.5280568893826537 1loss/SRqqt2/detection_loss‚0.6228752479033364 %loss/total_loss‚8.944490483376443 epoch‚10 trainer/global_step‚4149 /loss/FRt1/assignment_loss‚1.784623626152031 0loss/FRt2/assignment_loss‚0.9306541121030693 2loss/FBt1/assignment_loss‚0.008651509009883996 1loss/FBt2/assignment_loss‚0.03386605984035579 /loss/FRt2/detection_loss‚1.3739477389925459 1loss/SRqqt1/detection_loss‚0.9894302865261572 % _timestamp‚1.7843438774585388e+09 _step‚341 /loss/FRt1/detection_loss‚0.5949644432056913 /loss/FBt1/detection_loss‚0.9316232403119405 /loss/FBt2/detection_loss‚0.5736420728407163 _runtime‚3536.152204066(®  –‡ 2loss/SRqqt1/assignment_loss‚0.5721552571080617 2loss/SRqqt2/assignment_loss‚0.5280568893826537 1loss/SRqqt2/detection_loss‚0.6228752479033364 %loss/total_loss‚8.944490483376443 epoch‚10 trainer/global_step‚4149 /loss/FRt1/assignment_loss‚1.784623626152031 0loss/FRt2/assignment_loss‚0.9306541121030693 2loss/FBt1/assignment_loss‚0.008651509009883996 1loss/FBt2/assignment_loss‚0.03386605984035579 /loss/FRt2/detection_loss‚1.3739477389925459 1loss/SRqqt1/detection_loss‚0.9894302865261572 % _timestamp‚1.7843438774585388e+09 _step‚341 /loss/FRt1/detection_loss‚0.5949644432056913 /loss/FBt1/detection_loss‚0.9316232403119405 /loss/FBt2/detection_loss‚0.5736420728407163 _runtime‚3536.152204066Õ¨_Í¥“—‚(:ˆ ÉÚëҎऔ gpu.0.gpu‚95.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô,šq¿˜‚(:´ ÉÚëÒùЫ– disk.dm-1.out‚ 92.6015625 disk.drbd2165.out‚0 disk.sda.out‚ 92.6015625 disk.sda2.out‚0 cpu‚26.080875662341587 network.recv‚952740$ disk./.usageGB‚594.4290466308594 disk.rbd0.in‚0 disk.sda1.in‚0* disk./.usagePercent‚39.892632078587845 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9618.0791015625! disk.dm-0.out‚9618.0791015625 disk.sda3.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 7656.5703125 disk.rbd1.out‚ 9.01953125 network.sent‚ 6.403432e+06 disk.drbd2165.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚9618.0791015625# proc.memory.rssMB‚ 2716.51171875 disk.dm-0.in‚ 7656.5703125 disk.sda3.out‚ 92.6015625 disk.rbd1.in‚0$ memory_percent‚96.96698188781738' proc.memory.availableMB‚ 496.9296875 disk.sda1.out‚0) proc.memory.percent‚16.58027172088623 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 7656.5703125®0|Ê“™‚(:ˆ ØÚëÒľғ gpu.0.gpu‚65.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ä:XÁš‚(:¶ ØÚëÒÕ÷¥–* disk./.usagePercent‚39.892626446422085! disk.dm-0.out‚9624.9091796875 disk.nvme0n1.in‚ 7656.5703125 disk.drbd2165.in‚0) proc.memory.percent‚16.58027172088623 network.recv‚957220 disk.sda.out‚ 92.88671875$ memory_percent‚96.96676731109619 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 9.01953125 disk.sda1.out‚0$ disk./.usageGB‚594.4289627075195& disk.nvme0c0n1.out‚9624.9091796875 disk.dm-0.in‚ 7656.5703125$ disk.nvme0n1.out‚9624.9091796875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 92.88671875 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚26.07935199609661( proc.memory.availableMB‚ 496.96484375 network.sent‚ 6.46628e+06" disk.nvme0c0n1.in‚ 7656.5703125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 92.88671875 proc.cpu.threads‚17# proc.memory.rssMB‚ 2716.51171875Ã훂(:’ çÚëÒÞÅ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.166' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.43319999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¬XÉœ‚(:¾ çÚëÒžËå– network.sent‚ 6.471254e+06 disk.sda3.out‚ 93.40234375" disk.dm-0.out‚9633.51318359375! disk.nvme0n1.in‚ 7657.40234375 disk.sda1.out‚0 disk.sda.out‚ 93.40234375# proc.memory.rssMB‚ 2716.51171875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 9.14453125' disk.nvme0c0n1.out‚9633.51318359375$ memory_percent‚96.96524143218994* disk./.usagePercent‚39.892623118324146 disk.rbd0.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 497.21484375 network.recv‚960674 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 7657.40234375 cpu‚26.077902426828214 disk.drbd2165.out‚0% disk.nvme0n1.out‚9633.51318359375 disk.sda1.in‚0 disk.dm-1.out‚ 93.40234375) proc.memory.percent‚16.58027172088623$ disk./.usageGB‚594.4289131164551 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7657.40234375Þº2- _wandbruntime‚3575  _runtime‚3575&äˆYÆžÀ & lr-AdamW/pg1‚0.0004087192838440675 & lr-AdamW/pg2‚0.0004087192838440675 trainer/global_step‚4199 $ _timestamp‚1.784343916763179e+09 _runtime‚3575.456748112 _step‚342ˆtYËŸÅ trainer/global_step‚4199 $ _timestamp‚1.784343916763179e+09 _runtime‚3575.456748112 _step‚342 & lr-AdamW/pg1‚0.0004087192838440675 & lr-AdamW/pg2‚0.0004087192838440675Öš‹G‘2 - _wandbruntime‚3575  _runtime‚3575mX$à'¡ÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138923 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38883980 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9031992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642180518480896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148902 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160102400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4484067227136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634466720780288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33264920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637743814901760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3575.458110654 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33264922 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436686539776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145168041497088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140683885692928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚642180518480896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16391823 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345200640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7360081 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162722 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚4199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637743814901760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642180160102400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436686539776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7360078 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38874003 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38874003 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38883980 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4484155804160 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160102400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4484067227136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634466720780288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33264920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637743814901760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3575.458110654 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33264922 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436686539776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145168041497088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140683885692928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚642180518480896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16391823 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345200640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7360081 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162722 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚4199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637743814901760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642180160102400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436686539776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7360078 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38874003 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38874003 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38883980 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4484155804160 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634613231452160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38893046 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357827584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772144640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚637891055931392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883055 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33272552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMgÅLvonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413758976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚637891072970752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚140717295839232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888428 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145202384121856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165600 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33272554 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16395606 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9034052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚140717314451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699173888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357827584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4485178949632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843439173783011e+09 _runtime‚3576.072045115 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9033783 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Qà›sï&¥éM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699173888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357827584 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4485178949632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843439173783011e+09 _runtime‚3576.072045115 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9033783 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883055 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299066 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16395878 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145202493401088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634613231452160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38893046 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357827584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772144640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚637891055931392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883055 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33272552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846906 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413758976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚637891072970752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚140717295839232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888428 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145202384121856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165600 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33272554 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16395606 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9034052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚140717314451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ØyÕ22¦- _wandbruntime‚3576  _runtime‚3576qŽ/§‚ % _timestamp‚1.7843439173811677e+09 0loss/FRt1/assignment_loss‚1.7807196800621392 2loss/FBt2/assignment_loss‚0.008368544980848443 1loss/SRqqt1/detection_loss‚1.0745585369503192 _runtime‚3576.074447049 _step‚345 /loss/FRt1/detection_loss‚0.5909479688875424 .loss/FBt1/detection_loss‚1.006102274613161 trainer/global_step‚4199 2loss/SRqqt1/assignment_loss‚0.5542416407408902 2loss/SRqqt2/assignment_loss‚0.5319934545815191 /loss/FRt2/detection_loss‚1.3413647045689105 .loss/FBt2/detection_loss‚0.505289215440262 0loss/FRt2/assignment_loss‚0.8834078754069375 2loss/FBt1/assignment_loss‚0.003416932251304753 1loss/SRqqt2/detection_loss‚0.5950243573241896 %loss/total_loss‚8.875435185808024 epoch‚10ƒA{ ¨‡ 0loss/FRt2/assignment_loss‚0.8834078754069375 2loss/FBt1/assignment_loss‚0.003416932251304753 1loss/SRqqt2/detection_loss‚0.5950243573241896 %loss/total_loss‚8.875435185808024 epoch‚10 % _timestamp‚1.7843439173811677e+09 0loss/FRt1/assignment_loss‚1.7807196800621392 2loss/FBt2/assignment_loss‚0.008368544980848443 1loss/SRqqt1/detection_loss‚1.0745585369503192 _runtime‚3576.074447049 _step‚345 /loss/FRt1/detection_loss‚0.5909479688875424 .loss/FBt1/detection_loss‚1.006102274613161 trainer/global_step‚4199 2loss/SRqqt1/assignment_loss‚0.5542416407408902 2loss/SRqqt2/assignment_loss‚0.5319934545815191 /loss/FRt2/detection_loss‚1.3413647045689105 .loss/FBt2/detection_loss‚0.505289215440262Ùhs#æž©‚(:“ öÚëÒðœÚ” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.793' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.358600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆoÂ;ê‚(:¸ öÚëÒü' disk.nvme0c0n1.out‚9643.07177734375% disk.nvme0n1.out‚9643.07177734375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 93.57421875 disk.dm-0.in‚ 7658.3671875 disk.rbd1.out‚ 9.14453125' proc.memory.availableMB‚ 496.5703125 network.sent‚ 6.480024e+06 disk.dm-1.in‚0$ disk./.usageGB‚594.4290161132812 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 7658.3671875 disk.sda.in‚0 disk.sda.out‚ 93.57421875 disk.nvme0n1.in‚ 7658.3671875 disk.rbd1.in‚0) disk./.usagePercent‚39.89263003052756$ memory_percent‚96.96917533874512 cpu‚26.07620605894574 network.recv‚971350 disk.sda3.out‚ 93.57421875# proc.memory.rssMB‚ 2716.51171875" disk.dm-0.out‚9643.07177734375 disk.sda2.in‚0) proc.memory.percent‚16.58027172088623 proc.cpu.threads‚17JÅ‘«‚(:† …ÛëÒÄÆÙ“ gpu.0.gpu‚88.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í†!¿¬‚(:´ …ÛëÒóÜ´• disk.sda2.out‚0 disk.rbd1.out‚ 9.20703125 disk.dm-1.out‚ 93.9453125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7754.4296875 disk.rbd1.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 7754.4296875) disk./.usagePercent‚39.89264923109263 disk.dm-1.in‚0 disk.dm-0.in‚ 7754.4296875 disk.sda.out‚ 93.9453125$ memory_percent‚96.96786403656006 network.sent‚ 6.54293e+06$ disk./.usageGB‚594.4293022155762 disk.sda.in‚0( proc.memory.availableMB‚ 496.78515625 network.recv‚975971 disk.sda2.in‚0% disk.nvme0n1.out‚9647.45751953125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.58027172088623" disk.dm-0.out‚9647.45751953125# proc.memory.rssMB‚ 2716.51171875 cpu‚26.0747986138959 disk.sda3.out‚ 93.9453125 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚9647.45751953125î+W ­‚(:„ ”ÛëÒÐúô“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á;í+É®‚(:¾ ”ÛëÒ……Ä–' disk.nvme0c0n1.out‚9657.75146484375 disk.drbd2165.out‚0 network.sent‚ 6.547881e+06 disk.sda3.in‚0 disk.rbd1.out‚ 9.20703125% disk.nvme0n1.out‚9657.75146484375 proc.cpu.threads‚17* disk./.usagePercent‚39.892683536102226 disk.dm-1.in‚0 disk.dm-1.out‚ 94.18359375( proc.memory.availableMB‚ 496.52734375 cpu‚26.073307697243816$ memory_percent‚96.96943759918213 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 7755.26171875 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 7755.26171875# proc.memory.rssMB‚ 2716.51171875$ disk./.usageGB‚594.4298133850098 disk.sda1.out‚0 disk.sda.out‚ 94.18359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚9657.75146484375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 7755.26171875) proc.memory.percent‚16.58027172088623 network.recv‚978855 disk.sda3.out‚ 94.18359375&–“±2¯- _wandbruntime‚3615  _runtime‚3615* ­Ç°Á & lr-AdamW/pg1‚0.0004076840862783984 & lr-AdamW/pg2‚0.0004076840862783984 trainer/global_step‚4249 % _timestamp‚1.7843439568567173e+09 _runtime‚3615.550417363 _step‚346Ž5Ì±Æ _step‚346 & lr-AdamW/pg1‚0.0004076840862783984 & lr-AdamW/pg2‚0.0004076840862783984 trainer/global_step‚4249 % _timestamp‚1.7843439568567173e+09 _runtime‚3615.550417363ÚÙ¾`W2²- _wandbruntime‚3615  _runtime‚3615ÐZ:Ïà'³ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚347 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4535312351232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907085824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280092 trainer/global_step‚4249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382041600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146890635970560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843439568579354e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040702464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487040702464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649593265464320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382041600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚3615.551667503é ½Ný´ßO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146890635970560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843439568579354e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040702464 @:DeviceStatsMonitor.on_train_batch_start/n¸«è%um_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487040702464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649593265464320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382041600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚3615.551667503 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚347 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4535312351232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907085824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280092 trainer/global_step‚4249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382041600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ÛRcª{2µ- _wandbruntime‚3616  _runtime‚3616‘ PÓê&¶äM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053329408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991659 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚649741519128064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337507 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347498 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33654154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519128064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645253124452352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146925084876288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚3616.165397021 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645253107412992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4536332498432 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394675712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347498 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053329408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142388733765632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991659 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649741160742400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16587398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4481200333814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146924979791360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394675712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9138325 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚641938765045760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394675712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347498 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053329408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142388733765632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991659 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649741160742400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16587398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4481200333814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146924979791360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001652 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394675712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9138325 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚641938765045760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053329408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991659 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚649741519128064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337507 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347498 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33654154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519128064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438ÜЗæ2¸- _wandbruntime‚3616  _runtime‚3616õà5b¹ÿ /loss/FRt2/detection_loss‚1.3750154999420423 trainer/global_step‚4249 # _timestamp‚1.78434395747453e+09 0loss/FRt2/assignment_loss‚0.8911449392889715 2loss/SRqqt2/assignment_loss‚0.5215668977061564 1loss/SRqqt1/detection_loss‚0.9213470984305743 1loss/SRqqt2/detection_loss‚0.6615869297143997 .loss/FBt1/detection_loss‚0.887882180718912 /loss/FBt2/detection_loss‚0.6756806632056832 %loss/total_loss‚8.969587656003219 epoch‚10 _step‚349 /loss/FRt1/detection_loss‚0.5704308062553083 _runtime‚3616.167846044 0loss/FRt1/assignment_loss‚1.7291158343299726 2loss/SRqqt1/assignment_loss‚0.6905022971997303 0loss/FBt1/assignment_loss‚0.0320719446252053 2loss/FBt2/assignment_loss‚0.013242564586262063«5œ§ º„ 1loss/SRqqt1/detection_loss‚0.9213470984305743 1loss/SRqqt2/detection_loss‚0.6615869297143997 .loss/FBt1/detection_loss‚0.887882180718912 /loss/FBt2/detection_loss‚0.6756806632056832 %loss/total_loss‚8.969587656003219 epoch‚10 _step‚349 /loss/FRt1/detection_loss‚0.5704308062553083 _runtime‚3616.167846044 0loss/FRt1/assignment_loss‚1.7291158343299726 2loss/SRqqt1/assignment_loss‚0.6905022971997303 0loss/FBt1/assignment_loss‚0.0320719446252053 2loss/FBt2/assignment_loss‚0.013242564586262063 /loss/FRt2/detection_loss‚1.3750154999420423 trainer/global_step‚4249 # _timestamp‚1.78434395747453e+09 0loss/FRt2/assignment_loss‚0.8911449392889715 2loss/SRqqt2/assignment_loss‚0.5215668977061564Ýmb­C’»‚(:‡ £ÛëÒÑ©Ò“ gpu.0.gpu‚44.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ü½¼‚(:² £Ûëқ⿖ disk.rbd0.in‚0) proc.memory.percent‚16.58027172088623 disk.rbd1.in‚0 disk.drbd2165.in‚0 network.recv‚982929 disk.dm-1.in‚0# proc.memory.rssMB‚ 2716.51171875$ disk.nvme0n1.out‚9673.3662109375 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚9673.3662109375 disk.sda.in‚0 disk.sda.out‚ 96.1796875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 7756.2265625! disk.dm-0.out‚9673.3662109375 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.07170934283212 proc.cpu.threads‚17) disk./.usagePercent‚39.89330486638782 disk.nvme0n1.in‚ 7756.2265625 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚594.4390716552734 disk.sda3.in‚0 disk.sda3.out‚ 96.1796875 disk.dm-1.out‚ 96.1796875$ memory_percent‚96.97041511535645' proc.memory.availableMB‚ 496.3671875 disk.dm-0.in‚ 7756.2265625 network.sent‚ 6.553391e+06 disk.rbd1.out‚ 9.26953125… ]Z7½‚(:“ ²ÛëÒ• ì“ gpu.0.gpu‚90.0 gpu.0.Âýþgmemory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê`cȺ¾‚(:¯ ²ÛëÒЋ¶– disk.sda3.out‚ 96.4453125) proc.memory.percent‚16.58027172088623 proc.cpu.threads‚17$ disk./.usageGB‚594.4348564147949 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 9.26953125 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 96.4453125 disk.sda.in‚0" memory_percent‚96.968674659729# proc.memory.rssMB‚ 2716.51171875 network.recv‚994782 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 7756.2265625# disk.nvme0n1.out‚9685.185546875( proc.memory.availableMB‚ 496.65234375 network.sent‚ 6.619752e+06* disk./.usagePercent‚39.893021978062485 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 7756.2265625% disk.nvme0c0n1.out‚9685.185546875 disk.sda.out‚ 96.4453125 disk.sda3.in‚0 disk.dm-0.in‚ 7756.2265625 disk.dm-0.out‚9685.185546875 disk.drbd2165.out‚0 cpu‚26.07026069995793䋹_2¿- _wandbruntime‚3655  _runtime‚3655à‚ô»ÇÀÁ & lr-AdamW/pg1‚0.0004066371256121165 & lr-AdamW/pg2‚0.0004066371256121165 trainer/global_step‚4299 % _timestamp‚1.7843439967709537e+09 _runtime‚3655.464518013 _step‚3509gsIÌÁÆ & lr-AdamW/pg1‚0.0004066371256121165 & lr-AdamW/pg2‚0.0004066371256121165 trainer/global_step‚4299 % _timestamp‚1.7843439967709537e+09 _runtime‚3655.464518013 _step‚350Þ—f(\2Â- _wandbruntime‚3655  _runtime‚3655QCÍÒß'ÃÙO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736204288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652467934904320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148612989848064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34028120 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077543424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240323 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652467917864960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚653648951323738 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7534581 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654069248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077543424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649117787967488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 % _timestamp‚1.7843439967720568e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736204288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657005654069248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012447744 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9240055 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚351 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811027 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16774633 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚657006012447744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148613094940160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652467934904320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39792884 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782907 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39792884 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16774904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4586333394432 trainer/global_step‚4299œ?(cä'ÄÞO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012447744 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9240055 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚351 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811027 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16774633 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚657006012447744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148613094940160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652467934904320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39792884 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782907 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39792884 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16774904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4586333394432 trainer/global_step‚4299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736204288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652467934904320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148612989848064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34028120 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077543424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240323 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652467917864960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚653648951323738 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7534581 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654069248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077543424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649117787967488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 % _timestamp‚1.7843439967720568e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736204288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657005654069248ß»âp2Å- _wandbruntime‚3656  _runtime‚3656ò8ªyê&ÆäM KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39791959 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538748831232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39801950 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266111488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801950 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39791959 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4587261146112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652615158894592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370724 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16778439 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652615175933952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚4299 _runtime‚3656.081310422 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4587351813120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827711 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090177536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144060171692032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148647542117376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049444[Öß¼ÇéM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144060171692032 4.DeviceStatsMonitor.on_train_batch_end/nu£Ø°3%m_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148647542117376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049444 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39791959 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538748831232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39801950 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266111488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801950 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39791959 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4587261146112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652615158894592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370724 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16778439 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652615175933952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚4299 _runtime‚3656.081310422 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4587351813120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827711 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090177536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169àžO®2È- _wandbruntime‚3656  _runtime‚3656iFÑAÉ€ 0loss/FRt2/assignment_loss‚0.8784884096017886 1loss/FBt1/assignment_loss‚0.06328131228044083 1loss/FBt2/assignment_loss‚0.02723236367498849 1loss/SRqqt1/detection_loss‚0.9137322844310415 1loss/SRqqt2/detection_loss‚0.6150114670161364 _step‚353 0loss/FRt1/assignment_loss‚1.8242433748329305 .loss/FRt2/detection_loss‚1.366049728575824 .loss/FBt1/detection_loss‚0.800171391831504 /loss/FBt2/detection_loss‚0.3787081829763742 epoch‚10 % _timestamp‚1.7843439973901744e+09 2loss/SRqqt2/assignment_loss‚0.5625197088049227 %loss/total_loss‚8.518522712277115 trainer/global_step‚4299 _runtime‚3656.083457715 2loss/SRqqt1/assignment_loss‚0.5356465289916723 /loss/FRt1/detection_loss‚0.5534379592594896øyQµ Ê… epoch‚10 % _timestamp‚1.7843439973901744e+09 2loss/SRqqt2/assignment_loss‚0.5625197088049227 %loss/total_loss‚8.518522712277115 trainer/global_step‚4299 _runtime‚3656.083457715 2loss/SRqqt1/assignment_loss‚0.5356465289916723 /loss/FRt1/detection_loss‚0.5534379592594896 0loss/FRt2/assignment_loss‚0.8784884096017886 1loss/FBt1/assignment_loss‚0.06328131228044083 1loss/FBt2/assignment_loss‚0.02723236367498849 1loss/SRqqt1/detection_loss‚0.9137322844310415 1loss/SRqqt2/detection_loss‚0.6150114670161364 _step‚353 0loss/FRt1/assignment_loss‚1.8242433748329305 .loss/FRt2/detection_loss‚1.366049728575824 .loss/FBt1/detection_loss‚0.800171391831504 /loss/FBt2/detection_loss‚0.3787081829763742áǵ,rœË‚(:‘ ÁÛëÒ¡Ú±“ gpu.0.gpu‚84.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.960000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø·£ ²Ì‚(:§ ÁÛëÒŒ¨˜˜ disk.drbd2165.out‚0 disk.sda.out‚ 96.86328125 disk.sda2.out‚0 disk.rbd1.out‚ 9.33203125$ memory_percent‚96.96733951568604 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚7835.5 disk.nvme0n1.in‚7835.5 disk.rbd1.in‚0 network.recv‚998602 disk.rbd0.in‚0 disk.dm-0.in‚7835.5 disk.sda.in‚0) disk./.usagePercent‚39.89299176917345 disk.sda1.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2716.51171875 cpu‚26.06886481223622 network.sent‚ 6.624721e+06 disk.sda1.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚9693.33251953125 disk.sda3.out‚ 96.86328125 disk.sda3.in‚0( proc.memory.availableMB‚ 496.87109375$ disk./.usageGB‚594.4344062805176 disk.dm-1.out‚ 96.86328125' disk.nvme0c0n1.out‚9693.33251953125" disk.dm-0.out‚9693.33251953125) proc.memory.percent‚16.58027172088623 disk.sda2.in‚0 proc.cpu.threads‚17†Ö©C“Í‚(:ˆ ÐÛëÒ̵ٕ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݬÈLÊ΂(:¿ ÐÛëÒ‹Üÿ–' disk.nvme0c0n1.out‚9703.97705078125" disk.dm-0.out‚9703.97705078125 disk.sda.in‚0% disk.nvme0n1.out‚9703.97705078125 cpu‚26.06752325468022 disk.rbd0.out‚0$ disk./.usageGB‚594.4347915649414 disk.dm-1.out‚ 97.19140625# disk.nvme0c0n1.in‚ 7836.46484375 disk.sda3.out‚ 97.19140625) proc.memory.percent‚16.58027172088623 network.sent‚ 6.68967e+06 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 7836.46484375 disk.sda3.in‚0! disk.nvme0n1.in‚ 7836.46484375 disk.rbd1.out‚ 9.33203125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 97.19140625 disk.sda2.out‚0% proc.memory.availableMB‚ 496.71875* disk./.usagePercent‚39.893017625934405 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.96826934814453 network.recv‚ 1.004042e+06 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2716.51171875 proc.cpu.threads‚17w“ó“Ï‚(:ˆ ßÛëÒÒ’²“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†þé%ÊЂ(:¿ ßÛëÒƒ€Å– disk.rbd1.out‚ 9.39453125) proc.memory.percent‚16.58027172088623 network.recv‚ 1.007751e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 7836.46484375 cpu‚26.06611503355077 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 7836.46484375! disk.nvme0n1.in‚ 7836.46484375 disk.rbd0.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚9706.70361328125 disk.rbd1.in‚0( proc.memory.availableMB‚ 496.70703125$ disk./.usageGB‚594.4346733093262 disk.dm-1.out‚ 97.5546875 disk.sda.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2716.51171875 proc.cpu.threads‚17) disk./.usagePercent‚39.89300968970085 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚9706.70361328125$ memory_percent‚96.96834087371826 disk.sda1.out‚0% disk.nvme0n1.out‚9706.70361328125 disk.sda3.out‚ 97.5546875 network.sent‚ 6.692744e+06 disk.sda.out‚ 97.5546875 disk.sda2.in‚0 disk.sda2.out‚0_°2Ñ- _wandbruntime‚3695  _runtime‚36956qtÄÉÒà ' lr-AdamW/pg1‚0.00040557846908954913 ' lr-AdamW/pg2‚0.00040557846908954913 trainer/global_step‚4349 % _timestamp‚1.7843440368502176e+09 _runtime‚3695.543808907 _step‚354¯”JÎÓÈ ' lr-AdamW/pg1‚0.00040557846908954913 ' lr-AdamW/pg2‚0.00040557846908954913 trainer/global_step‚4349 % _timestamp‚1.7843440368502176e+09 _runtime‚3695.543808907 _step‚354âÃÊ2Ô- _wandbruntime‚3695  _runtime‚3695°?¯à'ÕÚO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237359 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401052672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3695.545062173 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4588773045248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588431706112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843440368513553e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743814996 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7621831 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4637418595840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9344285 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659829986385920 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150335618067968 _step‚355 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401052672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773045248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659829969346560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4637321630208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661025101404246 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16965848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431706112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74657058 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150335502490112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16966116 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169vH 7ÖßO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚4349 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9344020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145698180859904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150335618067968 _step‚355 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401052672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773045248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659829969346560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4637321630208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661025101404246 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16965848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431706112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74657058 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150335502490112 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16966116 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237359 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401052672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3695.545062173 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4588773045248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588431706112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843440368513553e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743814996 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocateꚨ® d‚7621831 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4637418595840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9344285 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659829986385920 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9346374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145731609618432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664567013094912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚659977210376192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚4349 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚356 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673756 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150369947331584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656589849272320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663763 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307630 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4589785679360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16969674 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589444333056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34417352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664566654709248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246411 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4638337713152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9346101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4582773857632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256402 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661172265540590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991°*¶ï&ÙéM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661172265540590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664567013094912 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚659977227415552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4638434671616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444333056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40246411 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145731628230656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4589785679360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚3696.161237696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16969950 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9346374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145731609618432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664567013094912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚659977210376192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚4349 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚356 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673756 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150369947331584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656589849272320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663763 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307630 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4589785679360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16969674 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589444333056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34417352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664566654709248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246411 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4638337713152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9346101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4582773857632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256402äE‰nÈ2Ú- _wandbruntime‚3696  _runtime‚3696ì+À› Û„ /loss/FRt2/detection_loss‚1.3448035688654225 1loss/SRqqt1/detection_loss‚0.8847739102399063 %loss/total_loss‚8.888376648342902 % _timestamp‚1.7843440374692197e+09 _step‚357 0loss/FRt1/assignment_loss‚1.7506979746387359 0loss/FRt2/assignment_loss‚1.0959345421070608 2loss/SRqqt1/assignment_loss‚0.6296440881029556 2loss/SRqqt2/assignment_loss‚0.5698762977311886 3loss/FBt1/assignment_loss‚0.0024642571094987383 2loss/FBt2/assignment_loss‚0.022193335101043328 /loss/FBt1/detection_loss‚0.8521440050775005 /loss/FBt2/detection_loss‚0.5369344891704343 trainer/global_step‚4349 .loss/FRt1/detection_loss‚0.568457969251505 1loss/SRqqt2/detection_loss‚0.6304522109476496 epoch‚10 _runtime‚3696.162523833<ºáY܉ /loss/FBt2/detection_loss‚0.5369344891704343 trainer/global_step‚4349 .loss/FRt1/detection_loss‚0.568457969251505 1loss/SRqqt2/detection_loss‚0.6304522109476496 epoch‚10 _runtime‚3696.162523833 /loss/FRt2/detection_loss‚1.3448035688654225 1loss/SRqqt1/detection_loss‚0.8847739102399063 %loss/total_loss‚8.888376648342902 % _timestamp‚1.7843440374692197e+09 _step‚357 0loss/FRt1/assignment_loss‚1.7506979746387359 0loss/FRt2/assignment_loss‚1.0959345421070608 2loss/SRqqt1/assignment_loss‚0.6296440881029556 2loss/SRqqt2/assignment_loss‚0.5698762977311886 3loss/FBt1/assignment_loss‚0.0024642571094987383 2loss/FBt2/assignment_loss‚0.022193335101043328 /loss/FBt1/detection_loss‚0.8521440050775005å!û–žÝ‚(:“ îÛëÒ‚­¯” gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚245.379' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.075799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕžÀ¢¼Þ‚(:± îÛëÒ߃—& disk.nvme0c0n1.out‚9715.8017578125$ proc.memory.availableMB‚496.4375) proc.memory.percent‚16.58027172088623 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2716.51171875 network.recv‚ 1.0112e+06$ disk./.usageGB‚594.4347152709961 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 7837.296875) disk./.usagePercent‚39.89301250578372 disk.rbd1.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚9715.8017578125 disk.sda.out‚ 97.8046875 disk.sda1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚9715.8017578125 disk.dm-1.out‚ 97.8046875 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 7837.296875 cpu‚26.064570108149255 disk.sda2.out‚0 disk.nvme0n1.in‚ 7837.296875$ memory_percent‚96.96998596191406 disk.rbd1.out‚ 9.39453125 network.sent‚ 6.697575e+06 disk.sda3.out‚ 97.8046875 disk.rbd0.in‚0óE¤žß‚(:“ ýÛëÒ¨öæ“ gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.906600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06y«ÜÀà‚(:µ ýÛëÒØÜÜ• network.sent‚ 6.760589e+06 disk.dm-0.out‚ 9724.25390625! proc.memory.rssMB‚ 2716.515625) proc.memory.percent‚16.58029556274414 disk.sda1.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 496.3203125 disk.rbd0.out‚0 disk.sda.out‚ 98.07421875$ memory_percent‚96.97070121765137 proc.cpu.threads‚17 network.recv‚ 1.016369e+06$ disk./.usageGB‚594.4340896606445" disk.nvme0n1.out‚ 9724.25390625 disk.sda3.out‚ 98.07421875 disk.rbd1.out‚ 9.51953125 disk.sda2.out‚0 cpu‚26.063129919997614 disk.rbd0.in‚0 disk.dm-1.out‚ 98.07421875 disk.dm-0.in‚ 7928.2734375 disk.sda2.in‚0 disk.nvme0n1.in‚ 7928.2734375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 9724.25390625 disk.sda.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚39.892970520548104 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 7928.2734375èa¨‘á‚(:† ŒÜëÒ« •• gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YĬXÄâ‚(:¹ ŒÜëÒðòœ– disk.drbd2165.in‚0$ disk./.usageGB‚594.4342231750488' disk.nvme0c0n1.out‚9731.57275390625 disk.sda2.in‚0 disk.dm-0.in‚ 7928.2734375" disk.dm-0.out‚9731.57275390625( proc.memory.availableMB‚ 496.57421875 disk.sda3.out‚ 98.2265625 disk.rbd0.out‚0 disk.dm-1.out‚ 98.2265625 disk.sda2.out‚0% disk.nvme0n1.out‚9731.57275390625) proc.memory.percent‚16.58029556274414 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 7928.2734375 disk.sda.in‚0# memory_percent‚96.9691514968872 network.sent‚ 6.765526e+06) disk./.usagePercent‚39.89297948081181 disk.rbd1.out‚ 9.51953125! proc.memory.rssMB‚ 2716.515625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 98.2265625 disk.nvme0n1.in‚ 7928.2734375 cpu‚26.06180827867258 network.recv‚ 1.019724e+06 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0ˆ[ó2ã- _wandbruntime‚3735  _runtime‚3735“aÇäÁ & lr-AdamW/pg1‚0.0004045081847062264 & lr-AdamW/pg2‚0.0004045081847062264 trainer/global_step‚4399 % _timestamp‚1.7843440769235275e+09 _runtime‚3735.617091567 _step‚358F«s¢ÌåÆ _runtime‚3735.617091567 _step‚358 & lr-AdamW/pg1‚0.0004045081847062264 & lr-AdamW/pg2‚0.0004045081847062264 trainer/global_step‚4399 % _timestamp‚1.7843440769235275e+09æÂXP2æ- _wandbruntime‚3735  _runtime‚3735„ÿ¶z çÚO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506414592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663768855154688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484754 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895504 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7709078 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9448000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127207936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506414592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3735.618500897 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148036096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688598430720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688516145152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701788 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088`Êäòå'èßO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7709078 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9448000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127207936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506414592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3735.618500897 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148036096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688598430720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688516145152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701788 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506414592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663768855154688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484754 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895504çruÝ2é- _wandbruntime‚3736  _runtime‚3736<–‹è&êâM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671979401692672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17161104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34798954 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4640481181184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40700863 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17160840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152092606503424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794538 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139834880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9450017 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689630751232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671979760078336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760078336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40710854 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚663915382865920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499815 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663915365826560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139834880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34798952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9450278 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481181184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4689558958592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210460 _runtime‚3736.231986358 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147403066157056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40700863 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34798952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499815 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710854 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17160840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152092606503424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794538 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139834880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9450017 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689630751232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671979760078336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760078336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40710854 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚663915382865920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499815 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663915365826560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139834880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34798952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9450278 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481181184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4689558958592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train¼Oóüu_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210460 _runtime‚3736.231986358 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147403066157056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40700863 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34798952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499815 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710854 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826èëÞ2ì- _wandbruntime‚3736  _runtime‚3736Ü©Ní‚ 0loss/FRt2/assignment_loss‚0.7769567102827925 _step‚361 2loss/SRqqt1/assignment_loss‚0.6402475268263234 2loss/SRqqt2/assignment_loss‚0.5632762339653206 /loss/FRt1/detection_loss‚0.5689480119738958 /loss/FBt1/detection_loss‚0.7900483089725714 trainer/global_step‚4399 % _timestamp‚1.7843440775411108e+09 _runtime‚3736.234425421 0loss/FRt1/assignment_loss‚1.7348109359764934 1loss/FBt2/assignment_loss‚0.00385928110200291 /loss/FRt2/detection_loss‚1.3617719167144968 1loss/SRqqt1/detection_loss‚0.8685446240595498 0loss/SRqqt2/detection_loss‚0.675669493238055 %loss/total_loss‚8.455293106002454 epoch‚10 2loss/FBt1/assignment_loss‚0.009639190374932011 /loss/FBt2/detection_loss‚0.4615208725160216q›øï î‡ /loss/FBt1/detection_loss‚0.7900483089725714 trainer/global_step‚4399 % _timestamp‚1.7843440775411108e+09 _runtime‚3736.234425421 0loss/FRt1/assignment_loss‚1.7348109359764934 1loss/FBt2/assignment_loss‚0.00385928110200291 /loss/FRt2/detection_loss‚1.3617719167144968 1loss/SRqqt1/detection_loss‚0.8685446240595498 0loss/SRqqt2/detection_loss‚0.675669493238055 %loss/total_loss‚8.455293106002454 epoch‚10 2loss/FBt1/assignment_loss‚0.009639190374932011 /loss/FBt2/detection_loss‚0.4615208725160216 0loss/FRt2/assignment_loss‚0.7769567102827925 _step‚361 2loss/SRqqt1/assignment_loss‚0.6402475268263234 2loss/SRqqt2/assignment_loss‚0.5632762339653206 /loss/FRt1/detection_loss‚0.5689480119738958é;_Tžï‚(:“ ›ÜëÒ…¦” gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚166.474' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.294799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü˜Èð‚(:½ ›ÜëÒØÇã– disk.sda2.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2720.015625 cpu‚26.074853518241383 network.sent‚ 6.770398e+06( disk./.usagePercent‚39.8953398702775# disk.nvme0c0n1.in‚ 7929.10546875% disk.nvme0n1.out‚9756.68212890625 disk.rbd1.out‚ 9.58203125) proc.memory.percent‚16.60165786743164$ disk./.usageGB‚594.4693946838379 disk.sda3.in‚0' disk.nvme0c0n1.out‚9756.68212890625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda3.out‚ 98.5078125 network.recv‚ 1.023036e+06 disk.dm-0.in‚ 7929.10546875 disk.sda.out‚ 98.5078125 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 98.5078125$ memory_percent‚96.99313640594482! disk.nvme0n1.in‚ 7929.10546875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 492.64453125" disk.dm-0.out‚9756.68212890625 disk.sda2.in‚02Eª2ñ- _wandbruntime‚3753  _runtime‚3753|è5Š=ò„{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.707403838634491 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29086437821388245 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907447099685669 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3512616455554962 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.770508348941803 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026992198079824448 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030006755143404007 'particle/f_score‚0.6332888007164001 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02906665951013565 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8093624114990234 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030006755143404007 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32068687677383423 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7442757487297058 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7506251335144043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3358375132083893 particle/accuracy_1_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.671716034412384 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_4‚0.7927830219268799 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 trainer/global_step‚4399 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7329299449920654 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07370907813310623 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7017029523849487 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21223999559879303 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08028653264045715 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6688126921653748 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8079127073287964 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7540766596794128 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962279796600342 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9209206700325012 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7568091750144958 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2837100028991699 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8441231846809387 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06664849817752838 /particle/accuracy_3_of_4‚0.9912415742874146 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7023780345916748 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48508763313293457 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2604120671749115 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.060376815497875214 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2779184579849243 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6580525636672974 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07767213881015778 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3538280129432678 *jet/accuracy_1_of_3‚0.9218180775642395 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6934345960617065 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6634032726287842 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09619539231061935 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.748751699924469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2819709777832031 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45609423518180847 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.77569979429245 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944214820861816 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5832406282424927 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6650352478027344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6572281718254089 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2936510443687439 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.866515576839447 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3905036151409149 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863752722740173 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6849537491798401 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860622882843018 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04205073043704033 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09636373817920685 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6309046745300293 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03496886417269707 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5400000214576721 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11505252867937088 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8374941945075989 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.6174370050430298 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7132563591003418 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748202204704285 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039178889244794846 /particle/accuracy_4_of_4‚0.9467543363571167 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6478458046913147 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22047783434391022 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2837100028991699 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/accuracy‚0.846450924873352 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891313910484314 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2948688864707947 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8652873039245605 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.380041523157354 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6939824819564819 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7716494202613831 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942845106124878 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9782626628875732 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7504704594612122 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988226890563965 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992063641548157 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9389944672584534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 )jet/accuracy_2_of_3‚0.564619243144989 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7681595087051392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005535960197449 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 epoch‚10 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.652595579624176 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04631571099162102 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3932707607746124 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 % _timestamp‚1.7843440947220325e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028222182765603065 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953108429908752 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_2‚0.148319274187088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958719611167908 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029316706582903862 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953984022140503 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 _step‚362 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4778939187526703 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6650352478027344 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2744378447532654 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9090811610221863 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22259314358234406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4992465078830719 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.235097274184227 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8602797985076904 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8512999415397644 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7101865410804749 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7723292112350464 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8635708689689636 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6627504825592041 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7789575457572937 6validation_average_jet_accuracy‚0.3876623511314392 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6848746538162231 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21757347881793976 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22251571714878082 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21419617533683777 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2065688818693161 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.35207468271255493 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4808228015899658 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47594478726387024 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2245316207408905 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2278195321559906 *jet/accuracy_1_of_4‚0.9663072228431702 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.655363917350769 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22165991365909576 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2366274893283844 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8643803596496582 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21042212843894958 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8512999415397644 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8776679039001465 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3662727177143097 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 _runtime‚3753.416401785®ÅÃã#ó‰{ 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030006755143404007 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32068687677383423 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7442757487297058 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7506251335144043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3358375132083893 particle/accuracy_1_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.671716034412384 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_4‚0.7927830219268799 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 trainer/global_step‚4399 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7329299449920654 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07370907813310623 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7017029523849487 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21223999559879303 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08028653264045715 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6688126921653748 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8079127073287964 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7540766596794128 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962279796600342 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9209206700325012 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7568091750144958 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2837100028991699 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8441231846809387 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06664849817752838 /particle/accuracy_3_of_4‚0.9912415742874146 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7023780345916748 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48508763313293457 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2604120671749115 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.060376815497875214 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2779184579849243 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6580525636672974 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07767213881015778 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3538280129432678 *jet/accuracy_1_of_3‚0.9218180775642395 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6934345960617065 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6634032726287842 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09619539231061935 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.748751699924469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2819709777832031 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45609423518180847 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.77569979429245 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944214820861816 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5832406282424927 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6650352478027344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6572281718254089 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2936510443687439 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.866515576839447 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3905036151409149 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863752722740173 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6849537491798401 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860622882843018 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04205073043704033 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09636373817920685 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6309046745300293 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03496886417269707 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5400000214576721 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11505252867937088 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8374941945075989 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.6174370050430298 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7132563591003418 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748202204704285 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039178889244794846 /particle/accuracy_4_of_4‚0.9467543363571167 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6478458046913147 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22047783434391022 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2837100028991699 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/accuracy‚0.846450924873352 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891313910484314 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2948688864707947 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8652873039245605 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.380041523157354 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6939824819564819 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7716494202613831 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942845106124878 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9782626628875732 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7504704594612122 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988226890563965 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992063641548157 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9389944672584534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 )jet/accuracy_2_of_3‚0.564619243144989 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7681595087051392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005535960197449 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 epoch‚10 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.652595579624176 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SR(÷j?¬qqt*FBt/event_purity‚0.04631571099162102 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3932707607746124 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 % _timestamp‚1.7843440947220325e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028222182765603065 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953108429908752 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_2‚0.148319274187088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958719611167908 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029316706582903862 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953984022140503 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 _step‚362 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4778939187526703 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6650352478027344 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2744378447532654 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9090811610221863 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22259314358234406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4992465078830719 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.235097274184227 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8602797985076904 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8512999415397644 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7101865410804749 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7723292112350464 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8635708689689636 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6627504825592041 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7789575457572937 6validation_average_jet_accuracy‚0.3876623511314392 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6848746538162231 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21757347881793976 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22251571714878082 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21419617533683777 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2065688818693161 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.35207468271255493 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4808228015899658 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47594478726387024 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2245316207408905 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2278195321559906 *jet/accuracy_1_of_4‚0.9663072228431702 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.655363917350769 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22165991365909576 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2366274893283844 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8643803596496582 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21042212843894958 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8512999415397644 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8776679039001465 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3662727177143097 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 _runtime‚3753.416401785 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.707403838634491 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29086437821388245 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907447099685669 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3512616455554962 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.770508348941803 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026992198079824448 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030006755143404007 'particle/f_score‚0.6332888007164001 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02906665951013565 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8093624114990234 ("Purity/2FRt2SRqqt*FBt/event_purity‚0êbßMžô‚(:“ ªÜëÒÚÀÞ• gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œîòÏõ‚(:Ä ªÜëÒÚ¯ù˜ network.sent‚ 6.833238e+06 disk.rbd1.in‚0 disk.sda.out‚ 98.73046875 disk.drbd2165.in‚0 network.recv‚ 1.028255e+06 disk.rbd0.in‚0( proc.memory.availableMB‚ 742.93359375 disk.rbd1.out‚ 9.58203125" disk.nvme0c0n1.in‚ 7930.0703125# disk.dm-0.out‚10459.60302734375 cpu‚26.075580592116182 disk.sda2.out‚0$ memory_percent‚95.46549320220947 disk.drbd2165.out‚0* disk./.usagePercent‚39.945521955124214 disk.sda1.in‚0( disk.nvme0c0n1.out‚10459.60302734375# proc.memory.rssMB‚ 2720.46484375) proc.memory.percent‚16.60439968109131 disk.dm-1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 7930.0703125 disk.nvme0n1.in‚ 7930.0703125& disk.nvme0n1.out‚10459.60302734375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 98.73046875$ disk./.usageGB‚595.2171440124512 disk.rbd0.out‚0 disk.dm-1.out‚ 98.73046875 disk.sda.in‚0 proc.cpu.threads‚17Z K>žö‚(:“ ¹ÜëҪГ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÄÁÁ÷‚(:¶ ¹Üëңブ cpu‚26.074208557733687 proc.cpu.threads‚17) disk./.usagePercent‚39.89281077184674$ disk./.usageGB‚594.4317092895508 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 99.0546875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 8040.90625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 496.95703125 disk.nvme0n1.in‚ 8040.90625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2720.46484375' disk.nvme0c0n1.out‚11065.0458984375 disk.dm-0.in‚ 8040.90625 disk.rbd1.out‚ 9.64453125) proc.memory.percent‚16.60439968109131 network.recv‚ 1.03337e+06 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚96.96681499481201 disk.drbd2165.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚11065.0458984375 disk.rbd1.in‚0 network.sent‚ 6.890586e+06 disk.dm-1.out‚ 99.0546875" disk.dm-0.out‚11065.0458984375 disk.sda.out‚ 99.0546875 disk.rbd0.in‚0Îv‹Õ2ø- _wandbruntime‚3794  _runtime‚3794_ÿð³Ãù½ trainer/global_step‚4449 # _timestamp‚1.78434413623191e+09 _runtime‚ 3794.9254735 _step‚363 & lr-AdamW/pg1‚0.0004034263412045143 & lr-AdamW/pg2‚0.0004034263412045143æ9RAÈú & lr-AdamW/pg1‚0.0004034263412045143 & lr-AdamW/pg2‚0.0004034263412045143 trainer/global_step‚4449 # _timestamp‚1.78434413623191e+09 _runtime‚ 3794.9254735 _step‚363뺟2û- _wandbruntime‚3794  _runtime‚3794JzË°à'üÚO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚3794.926933226 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553256448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153814855327744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41216590 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194877952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9582512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520155 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194877952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9582512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520155 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35301218 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153849415731712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149090397571072 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536853 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681339806920192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9584872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41235633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9584624 _runtime‚3795.538980444 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843441368452516e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚676630879989760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230282 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225642 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚365 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909891072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41235633 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536853 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17385468 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708568544768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35301220 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17385217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448534528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153849310646784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708568544768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35301220 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17385217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448534528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153849310646784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35301218 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153849415731712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149090397571072 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536853 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681339806920192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9584872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41235633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9584624 _runtime‚3795.538980444 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843441368452516e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚676630879989760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230282 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225642 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚365 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909891072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41235633 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536853 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360íY’ùl2ƒ- _wandbruntime‚3795  _runtime‚3795l1…v „„ 1loss/SRqqt1/detection_loss‚0.9834210454202178 /loss/FRt2/detection_loss‚1.3284413196975948 1loss/SRqqt2/detection_loss‚0.6912989596392881 epoch‚11 % _timestamp‚1.7843441368496034e+09 _step‚366 2loss/SRqqt2/assignment_loss‚0.4859707176970209 3loss/FBt1/assignment_loss‚0.0025136944302157705 /loss/FBt2/detection_loss‚0.6441741008992092 2loss/SRqqt1/assignment_loss‚0.6404726266636024 /loss/FRt1/detection_loss‚0.5684095834550319 /loss/FBt1/detection_loss‚0.9166971081546669 $loss/total_loss‚8.98779011607979 trainer/global_step‚4449 _runtime‚3795.542879955 0loss/FRt1/assignment_loss‚1.6764905120662315 /loss/FRt2/assignment_loss‚1.046572910870115 3loss/FBt2/assignment_loss‚0.0033275370865965214„À B…‰ 0loss/FRt1/assignment_loss‚1.6764905120662315 /loss/FRt2/assignment_loss‚1.046572910870115 3loss/FBt2/assignment_loss‚0.0033275370865965214 1loss/SRqqt1/detection_loss‚0.9834210454202178 /loss/FRt2/detection_loss‚1.3284413196975948 1loss/SRqqt2/detection_loss‚0.6912989596392881 epoch‚11 % _timestamp‚1.7843441368496034e+09 _step‚366 2loss/SRqqt2/assignment_loss‚0.4859707176970209 3loss/FBt1/assignment_loss‚0.0025136944302157705 /loss/FBt2/detection_loss‚0.6441741008992092 2loss/SRqqt1/assignment_loss‚0.6404726266636024 /loss/FRt1/detection_loss‚0.5684095834550319 /loss/FBt1/detection_loss‚0.9166971081546669 $loss/total_loss‚8.98779011607979 trainer/global_step‚4449 _runtime‚3795.542879955F5ž†‚(:“ ×ÜëÒ¼¢ß“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º·FD͇‚(: ×ÜëÒÛªœ—# disk.dm-0.out‚11103.17138671875 disk.sda2.out‚0' proc.memory.availableMB‚ 496.9140625) disk./.usagePercent‚39.89270043259948 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚11103.17138671875 disk.sda.out‚ 102.796875 proc.cpu.threads‚17 disk.sda2.in‚0! disk.nvme0n1.in‚ 8041.73828125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8041.73828125 disk.sda.in‚0& disk.nvme0n1.out‚11103.17138671875 network.sent‚ 6.960571e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 102.796875 disk.rbd1.out‚ 9.70703125 disk.dm-0.in‚ 8041.73828125 network.recv‚ 1.041785e+06 cpu‚26.071332483280223 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚96.96707725524902# proc.memory.rssMB‚ 2720.46484375) proc.memory.percent‚16.60439968109131$ disk./.usageGB‚594.4300651550293 disk.dm-1.in‚0 disk.dm-1.out‚ 102.796875 disk.drbd2165.in‚0‰² §“ˆ‚(:ˆ æÜëÒج‹• gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¿_elj‚(:¼ æÜëÒ…亖 disk.dm-1.out‚ 104.55859375 disk.dm-0.in‚ 8042.703125* proc.memory.percent‚16.605329513549805 network.sent‚ 7.000363e+06 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 8042.703125$ disk./.usageGB‚594.4270553588867! disk.dm-0.out‚11119.509765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.6171875 disk.sda.in‚0 disk.nvme0n1.in‚ 8042.703125 disk.drbd2165.out‚0 network.recv‚ 1.046711e+06) disk./.usagePercent‚39.89249844265497 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 104.55859375 cpu‚26.069962576087008 proc.cpu.threads‚17& disk.nvme0c0n1.out‚11119.509765625 disk.dm-1.in‚0$ disk.nvme0n1.out‚11119.509765625$ memory_percent‚96.93121910095215' proc.memory.availableMB‚ 502.7890625 disk.sda.out‚ 104.55859375 disk.rbd1.out‚ 9.70703125xd9¼2Š- _wandbruntime‚3834  _runtime‚3834€ŠùTÉ‹Ã ' lr-AdamW/pg1‚0.00040233300806919933 ' lr-AdamW/pg2‚0.00040233300806919933 trainer/global_step‚4499 % _timestamp‚1.7843441762645047e+09 _runtime‚3834.958091692 _step‚367ÒxnWÎŒÈ trainer/global_step‚4499 % _timestamp‚1.7843441762645047e+09 _runtime‚3834.958091692 _step‚367 ' lr-AdamW/pg1‚0.00040233300806919933 ' lr-AdamW/pg2‚0.00040233300806919933S¦2- _wandbruntime‚3834  _runtime‚3834:—ôj‘"ŽØO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7886101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155537702774272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3834.959433375 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4758251419648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41681019 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17572512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784344176265708e+09 _step‚368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155537606070784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683845690441728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592758784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4809199331840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585084 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚4499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41671042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4809277423104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688604300239872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150728õ¶ M406738944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941861376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17572242 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680328602595328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0I5F[ã'ÝO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7886101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155537702774272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3834.959433375 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4758251419648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41681019 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17572512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784344176265708e+09 _step‚368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155537606070784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683845690441728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592758784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4809199331840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585084 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚4499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41671042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4809277423104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688604300239872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150728406738944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941861376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17572242 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680328602595328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088ð¬†ÀR2- _wandbruntime‚3835  _runtime‚3835ç醣ê&‘äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553903616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680094 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4810303919104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35682818 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4810215349248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264046592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264046592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17576025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310790 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843441768793278e+09 _step‚369 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372905 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195517952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9688455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688752553903616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155572050846720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721428 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683992948510720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372905 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155572158028800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680475096227840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41680094 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9688182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043590 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4759605392896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605392896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41690085 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4810303919104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35682818 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4810215349248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264046592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264046592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17576025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310790 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843441768793278e+09 _step‚369 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372905 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195517952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9688455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688752553903616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155572050846720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721428 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683992948510720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372905 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ñØñY2“- _wandbruntime‚3835  _runtime‚3835¿­s ”ƒ 2loss/FBt2/assignment_loss‚0.007571920973275763 /loss/FRt1/detection_loss‚0.5744868427759078 1loss/SRqqt1/detection_loss‚0.8896082182602284 _step‚370 /loss/FRt2/assignment_loss‚1.003285775059148 2loss/FBt1/assignment_loss‚0.006523775550245858 % _timestamp‚1.7843441768828475e+09 _runtime‚ 3835.57612935 0loss/FRt1/assignment_loss‚1.8021812483047446 2loss/SRqqt2/assignment_loss‚0.5068028464537128 /loss/FRt2/detection_loss‚1.2235295140982874 /loss/FBt1/detection_loss‚0.8538569338270463 %loss/total_loss‚8.607661997564971 2loss/SRqqt1/assignment_loss‚0.5874144848303476 1loss/SRqqt2/detection_loss‚0.6700901137900692 0loss/FBt2/detection_loss‚0.48231032364195786 epoch‚11 trainer/global_step‚4499&kÅýR•ˆ 2loss/FBt2/assignment_loss‚0.007571920973275763 /loss/FRt1/detection_loss‚0.5744868427759078 1loss/SRqqt1/detection_loss‚0.8896082182602284 _step‚370 /loss/FRt2/assignment_loss‚1.003285775059148 2loss/FBt1/assignment_loss‚0.006523775550245858 % _timestamp‚1.7843441768828475e+09 _runtime‚ 3835.57612øK<Ô¼935 0loss/FRt1/assignment_loss‚1.8021812483047446 2loss/SRqqt2/assignment_loss‚0.5068028464537128 /loss/FRt2/detection_loss‚1.2235295140982874 /loss/FBt1/detection_loss‚0.8538569338270463 %loss/total_loss‚8.607661997564971 2loss/SRqqt1/assignment_loss‚0.5874144848303476 1loss/SRqqt2/detection_loss‚0.6700901137900692 0loss/FBt2/detection_loss‚0.48231032364195786 epoch‚11 trainer/global_step‚4499òÞý²ž–‚(:“ õÜëÒ„äñ• gpu.0.gpu‚62.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F/0Ì—‚(:Á õÜëÒªÙÄ—& proc.memory.availableMB‚ 502.703125" proc.memory.rssMB‚ 2720.6171875 cpu‚26.06857779302515 network.recv‚ 1.050165e+06 proc.cpu.threads‚17 network.sent‚ 7.018249e+06' disk.nvme0c0n1.out‚11133.6123046875" disk.dm-0.out‚11133.6123046875 disk.sda.in‚0% disk.nvme0n1.out‚11133.6123046875 disk.rbd1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 8150.8046875 disk.dm-0.in‚ 8150.8046875 disk.sda.out‚ 105.08984375 disk.sda2.out‚0 disk.rbd1.out‚ 9.76953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 105.08984375 disk.dm-1.in‚0 disk.dm-1.out‚ 105.08984375* proc.memory.percent‚16.605329513549805* disk./.usagePercent‚39.892566284651544 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚594.4280662536621 disk.nvme0n1.in‚ 8150.8046875 disk.rbd0.out‚0$ memory_percent‚96.93174362182617,ð"€ž˜‚(:“ „ÝëÒ„¨ä“ gpu.0.gpu‚37.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ̓Ï™‚(:Ä „ÝëÒð¡–$ disk./.usageGB‚594.4306564331055( disk.nvme0c0n1.out‚11146.36572265625 disk.dm-0.in‚ 8150.8046875# disk.dm-0.out‚11146.36572265625 disk.sda.out‚ 107.23828125 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 9.76953125 network.sent‚ 7.067875e+06) disk./.usagePercent‚39.89274011376729 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 107.23828125" disk.nvme0c0n1.in‚ 8150.8046875$ memory_percent‚96.92983627319336 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚26.067107663214266& proc.memory.availableMB‚ 503.015625& disk.nvme0n1.out‚11146.36572265625" proc.memory.rssMB‚ 2720.6171875 disk.dm-1.in‚0 disk.sda.in‚0 network.recv‚ 1.054391e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 8150.8046875 disk.sda3.out‚ 107.23828125 proc.cpu.threads‚17* proc.memory.percent‚16.605329513549805(ÑXì“š‚(:ˆ “ÝëҢゕ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍI+¿›‚(:´ “ÝëÒªùÉ– disk.sda3.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.6171875$ disk./.usageGB‚594.4421157836914# disk.nvme0c0n1.in‚ 8151.76953125 disk.dm-0.out‚ 11163.34375( proc.memory.availableMB‚ 503.16796875* proc.memory.percent‚16.605329513549805 cpu‚26.065754922488914 proc.cpu.threads‚17 network.sent‚ 7.074118e+06 network.recv‚ 1.057631e+06) disk./.usagePercent‚39.89350916040015 disk.sda1.in‚0 disk.dm-1.out‚ 110.0703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 8151.76953125 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 110.0703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 110.0703125 disk.rbd0.in‚0 disk.nvme0n1.out‚ 11163.34375! disk.nvme0n1.in‚ 8151.76953125 disk.rbd1.out‚ 9.83203125$ memory_percent‚96.92890644073486 disk.sda1.out‚0" disk.nvme0c0n1.out‚ 11163.34375R—!Ç2œ- _wandbruntime‚3874  _runtime‚3874—Ÿ‹îÉà ' lr-AdamW/pg1‚0.00040122825552302575 ' lr-AdamW/pg2‚0.00040122825552302575 trainer/global_step‚4549 % _timestamp‚1.7843442162107022e+09 _runtime‚3874.904429602 _step‚371Ô¾PÎžÈ ' lr-AdamW/pg1‚0.00040122825552302575 ' lr-AdamW/pg2‚0.00040122825552302575 trainer/global_step‚4549 % _timestamp‚1.7843442162107022e+09 _runtime‚3874.904429602 _step‚371ó&à¸2Ÿ- _wandbruntime‚3874  _runtime‚3874§­Oà' ÚO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288260608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17763069 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36056786 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688844800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692455727076342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42135471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946921472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157260280416256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7973351 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 _step‚372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36056788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17762792 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946921472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157260280416256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7973351 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 _step‚372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36056788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17762792 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17763069 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36056786 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688844800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692455727076342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888ô²}%“2¢- _wandbruntime‚3875  _runtime‚3875?t{ À£ãM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391298 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36064418 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134546 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300887040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7975096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144537 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4861443155456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17766565 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9791734 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚373 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959548416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3875.520314358 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157294636398080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843442168264902e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208957 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801936 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530498 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300894720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265qsî&¤èM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391298 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36064418 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134546 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300887040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7975096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144537 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4861443155456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17766565 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9791734 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚373 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959548416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3875.520314358 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157294636398080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843442168264902e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208957 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801936 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530498 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300894720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198964 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1õI¼O‰2¥- _wandbruntime‚3875  _runtime‚3875ô<à ¦ƒ 1loss/SRqqt1/detection_loss‚0.9256107987513622 /loss/FBt1/detection_loss‚0.8503756126896902 % _timestamp‚1.7843442168295355e+09 0loss/FRt1/assignment_loss‚1.7234716519919262 2loss/SRqqt1/assignment_loss‚0.5566952181187121 2loss/SRqqt2/assignment_loss‚0.5435404226388348 2loss/FBt2/assignment_loss‚0.014351270974492451 /loss/FRt1/detection_loss‚0.6004013710693399 epoch‚11 0loss/FRt2/assignment_loss‚0.8449591474515646 1loss/SRqqt2/detection_loss‚0.6170592643517767 %loss/total_loss‚8.536744143934488 trainer/global_step‚4549 _runtime‚3875.522835889 _step‚374 2loss/FBt1/assignment_loss‚0.047498715975124534 .loss/FRt2/detection_loss‚1.190746302488711 /loss/FBt2/detection_loss‚0.6220343674329535Ü tÓ§ˆ 0loss/FRt1/assignment_loss‚1.7234716519919262 2loss/SRqqt1/assignment_loss‚0.5566952181187121 2loss/SRqqt2/assignment_loss‚0.5435404226388348 2loss/FBt2/assignment_loss‚0.014351270974492451 /loss/FRt1/detection_loss‚0.6004013710693399 epoch‚11 0loss/FRt2/assignment_loss‚0.8449591474515646 1loss/SRqqt2/detection_loss‚0.6170592643517767 %loss/total_loss‚8.536744143934488 trainer/global_step‚4549 _runtime‚3875.522835889 _step‚374 2loss/FBt1/assignment_loss‚0.047498715975124534 .loss/FRt2/detection_loss‚1.190746302488711 /loss/FBt2/detection_loss‚0.6220343674329535 1loss/SRqqt1/detection_loss‚0.9256107987513622 /loss/FBt1/detection_loss‚0.8503756126896902 % _timestamp‚1.7843442168295355e+09ö×®´‘¨‚(:† ¢ÝëÒ¹‹«” gpu.0.gpu‚75.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚212.53' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§¢ nÏ©‚(:Ä ¢ÝëÒÚÔÝ–& disk.nvme0n1.out‚11734.78955078125$ memory_percent‚96.92909717559814 disk.rbd1.out‚ 9.83203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2720.6171875 network.recv‚ 1.060903e+06' disk./.usagePercent‚39.975401362468 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11734.78955078125 proc.cpu.threads‚17 network.sent‚ 7.077141e+06$ disk./.usageGB‚595.6623687744141 disk.sda.out‚ 113.90234375 disk.dm-1.out‚ 113.90234375* proc.memory.percent‚16.605329513549805 disk.sda3.out‚ 113.90234375 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 8152.6015625 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚26.064478797933038 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 8152.6015625( proc.memory.availableMB‚ 503.13671875 disk.sda1.in‚0# disk.dm-0.out‚11734.78955078125 disk.nvme0n1.in‚ 8152.6015625¿á%|ª‚(:„ ±ÝëÒòØΓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|0OÉ«‚(:¾ ±ÝëÒšüï– disk.rbd0.out‚0# memory_percent‚96.9306468963623 disk.dm-0.in‚ 8236.43359375 disk.sda2.out‚0# disk.nvme0n1.out‚11815.04296875 disk.sda3.out‚ 116.54296875 disk.dm-1.out‚ 116.54296875 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 502.8828125* proc.memory.percent‚16.605329513549805 cpu‚26.063149003778328 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚597.5943489074707 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 9.89453125 network.sent‚ 7.139972e+06 network.recv‚ 1.066232e+06) disk./.usagePercent‚40.10505817023674# disk.nvme0c0n1.in‚ 8236.43359375 disk.sda1.in‚0 disk.dm-0.out‚11815.04296875% disk.nvme0c0n1.out‚11815.04296875 disk.sda.in‚0 disk.sda.out‚ 116.54296875! disk.nvme0n1.in‚ 8236.43359375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.6171875#Y2¬- _wandbruntime‚3914  _runtime‚3914ˆÎKÉ­Ã trainer/global_step‚4599 % _timestamp‚1.7843442562176526e+09 _runtime‚3914.911352745 _step‚375 ' lr-AdamW/pg1‚0.00040011215452218513 ' lr-AdamW/pg2‚0.000400112154522185133€å¥Î®È ' lr-AdamW/pg1‚0.00040011215452218513 ' lr-AdamW/pg2‚0.00040011215452218513 trainer/global_step‚4599 % _timestamp‚1.7843442562176526e+09 _runtime‚3914.911352745 _step‚375÷W.„I2¯- _wandbruntime‚3914  _runtime‚3914TÎwà'°ÚO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8060601 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154071301203968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4911557430784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚699831877156850 _step‚376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36438386 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42579946 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36438388 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42579946 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018332 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435828224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843442562188647e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17953014 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚699832234746100 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4852224413682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36438388 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9892413 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589923 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983762432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚4599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_Cò`Apool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589923 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983762432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚4599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36438388 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42579946 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018332 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435828224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224øka©…ž²‚(:“ ÀÝëÒ³½‹” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚282.041' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.408199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¨h̳‚(:Á ÀÝëÒæìÊ• disk.sda3.out‚ 118.83203125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda1.in‚0$ memory_percent‚96.93207740783691 network.recv‚ 1.069621e+06 disk.dm-0.in‚ 8237.3984375 disk.nvme0n1.in‚ 8237.3984375* proc.memory.percent‚16.605329513549805' disk.nvme0c0n1.out‚13615.9619140625' proc.memory.availableMB‚ 502.6484375" proc.memory.rssMB‚ 2720.6171875 cpu‚26.061785351249114 network.sent‚ 7.144916e+06# disk./.usageGB‚596.704158782959 disk.rbd1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚40.04531676406149" disk.nvme0c0n1.in‚ 8237.3984375 disk.sda.out‚ 118.83203125 disk.rbd1.out‚ 9.89453125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 118.83203125 disk.drbd2165.in‚0" disk.dm-0.out‚13615.9619140625 disk.sda2.out‚0% disk.nvme0n1.out‚13615.9619140625 disk.sda3.in‚0ã%(2´- _wandbruntime‚3915  _runtime‚3915¦ ¡ê&µäM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698717034434560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚377 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42588998 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17956582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598989 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703577689484800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467404 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4912575689728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045009 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703578047870464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4912485022720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154104729962496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882444 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17956848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655050240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045009 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843442568324108e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154104729962496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882444 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17956848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655050240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045009 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843442568324108e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚377 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42588998 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17956582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598989 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703577689484800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467404 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4912575689728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045009 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703578047870464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4912485022720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9995436 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4962687104000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18143552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707208027237358 trainer/global_step‚4649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36819986 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚705931861925888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541190144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36819988 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4903181450480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚705931844886528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337925120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚705931844886528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4910679264256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4962773583872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚702305203376128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182811648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚705931861925888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4910679264256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182811648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160705512714240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Š¶å'ÃßO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43044375 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18143284 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9995701 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚380 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4910337925120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9995436 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4962687104000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18143552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707208027237358 trainer/global_step‚4649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36819986 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚705931861925888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541190144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36819988 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4903181450480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚705931844886528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337925120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚705931844886528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4910679264256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4962773583872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚702305203376128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182811648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚705931861925888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4910679264256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182811648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160705512714240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43034398 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881061 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350552064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8149593 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚381 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881061 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18147382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706079102955520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18147129 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691898368 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4963719316992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794853888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871068 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160739967387136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4649 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871068 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053441 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843442967978132e+09 _runtime‚3955.491575427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436468224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691898368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360dÑ{» ÆèM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871068 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053441 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843442967978132e+09 _runtime‚3955.491575427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436468224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691898368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155776167888896 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707355906552314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043450 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710990794853888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4911350552064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881061 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350552064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8149593 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚381 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881061 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18147382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706079102955520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18147129 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691898368 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4963719316992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794853888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871068 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160739967387136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4649ýªLôi2Ç- _wandbruntime‚3955  _runtime‚3955^®5/È‚ .loss/FRt2/detection_loss‚1.270711056378405 /loss/FBt1/detection_loss‚0.6347824747984608 /loss/FBt2/detection_loss‚0.5128624535109493 % _timestamp‚1.7843442968018763e+09 3loss/SRqqt2/assignment_loss‚0.47891380724274063 2loss/FBt1/assignment_loss‚0.013572538875052942 1loss/SRqqt2/detection_loss‚0.6646930250248396 trainer/global_step‚4649 _runtime‚3955.495219438 0loss/FRt1/assignment_loss‚1.7899007380904046 2loss/SRqqt1/assignment_loss‚0.6154522608729117 1loss/FBt2/assignment_loss‚0.04037627288610703 /loss/FRt1/detection_loss‚0.5700992679298165 %loss/total_loss‚8.643402665521037 _step‚382 0loss/FRt2/assignment_loss‚0.9954275276777003 0loss/SRqqt1/detection_loss‚1.056611242233648 epoch‚11\á s ɇ 0loss/FRt1/assignment_loss‚1.7899007380904046 2loss/SRqqt1/assignment_loss‚0.6154522608729117 1loss/FBt2/assignment_loss‚0.04037627288610703 /loss/FRt1/detection_loss‚0.5700992679298165 %loss/total_loss‚8.643402665521037 _step‚382 0loss/FRt2/assignment_loss‚0.9954275276777003 0loss/SRqqt1/detection_loss‚1.056611242233648 epoch‚11 .loss/FRt2/detection_loss‚1.270711056378405 /loss/FBt1/detection_loss‚0.6347824747984608 /loss/FBt2/detection_loss‚0.5128624535109493 % _timestamp‚1.7843442968018763e+09 3loss/SRqqt2/assignment_loss‚0.47891380724274063 2loss/FBt1/assignment_loss‚0.013572538875052942 1loss/SRqqt2/detection_loss‚0.6646930250248396 trainer/global_step‚4649 _runtime‚3955.495219438þÞ’Ò·“Ê‚(:ˆ íÝëÒ†ÖŸ” gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ßK[·Ë‚(:¬ íÝëÒ´ã²– disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚13684.82958984375 disk.sda.out‚122.3125 disk.sda3.in‚0 disk.nvme0c0n1.in‚8354.875* disk./.usagePercent‚40.045106325868346& disk.nvme0n1.out‚13684.82958984375 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 7.254257e+06 network.recv‚ 1.0832e+06 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.93231582641602" proc.memory.rssMB‚ 2720.6171875 cpu‚26.057880928047993 disk.rbd1.out‚ 10.01953125 disk.dm-0.in‚8354.875 disk.nvme0n1.in‚8354.875 disk.sda3.out‚122.3125 disk.sda1.out‚0# disk.dm-0.out‚13684.82958984375& proc.memory.availableMB‚ 502.609375* proc.memory.percent‚16.605329513549805$ disk./.usageGB‚596.7010231018066 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚122.3125­>6g“Ì‚(:ˆ üÝëÒøçÍ“ gpu.0.gpu‚52.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚149.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NwûÁÍ‚(:¶ üÝëÒ•š¶—$ proc.memory.availableMB‚502.6875 disk.sda1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚13692.06005859375 disk.dm-0.in‚ 8355.26953125 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 8355.26953125 disk.rbd0.in‚0 disk.dm-1.out‚122.875# disk.nvme0c0n1.in‚ 8355.26953125* proc.memory.percent‚16.605329513549805$ disk./.usageGB‚596.7004165649414 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.out‚122.875" proc.memory.rssMB‚ 2720.6171875 network.recv‚ 1.087931e+06 network.sent‚ 7.317176e+06 disk.sda2.in‚0 cpu‚26.056526008692767( disk./.usagePercent‚40.0450656206704 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚122.875 proc.cpu.threads‚17 disk.rbd1.out‚ 10.01953125$ memory_percent‚96.93183898925781 disk.drbd2165.in‚0# disk.dm-0.out‚13692.06005859375& disk.nvme0n1.out‚13692.06005859375 disk.rbd1.in‚0ÿ ֓΂(:ˆ ‹ÞëÒˣܓ gpu.0.gpu‚51.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.604' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.5208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GtËÏ‚(:À ‹ÞëÒ„ëÛ–* disk./.usagePercent‚40.045072788881356 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 123.265625 disk.sda3.out‚ 123.265625$ memory_percent‚96.93052768707275( proc.memory.availableMB‚ 502.90234375 cpu‚26.05532925331072 network.recv‚ 1.091405e+06# disk.nvme0c0n1.in‚ 8355.70703125* proc.memory.percent‚16.605329513549805 proc.cpu.threads‚17$ disk./.usageGB‚596.7005233764648 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 123.265625 disk.rbd1.out‚ 10.08203125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚13699.3466796875 disk.sda.in‚0% disk.nvme0n1.out‚13699.3466796875" proc.memory.rssMB‚ 2720.6171875! disk.nvme0n1.in‚ 8355.70703125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 8355.70703125" disk.dm-0.out‚13699.3466796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 7.32219e+06=ÿb2Ð- _wandbruntime‚3994  _runtime‚3994z–ðhÇÑÁ _step‚383 & lr-AdamW/pg1‚0.0003978461946211155 & lr-AdamW/pg2‚0.0003978461946211155 trainer/global_step‚4699 % _timestamp‚1.7843443359619503e+09 _runtime‚3994.6562876453E‹ÌÒÆ _runtime‚3994.656287645 _step‚383 & lr-AdamW/pg1‚0.0003978461946211155 & lr-AdamW/pg2‚0.0003978461946211155 trainer/global_step‚4699 % _timestamp‚1.7843443359619503e+09ÿIof(2Ó- _wandbruntime‚3994  _runtime‚3994ÿ¥/%á'ÔÛO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714584534907116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953457387498 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709630719930368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162428118383104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162427994416640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929795072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5013835972096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3994.656516457 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37201588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033426944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288173568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10098626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953457387498 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709630719930368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162428118383104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162427994416640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929795072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactiNqñ¤ve_split_bytes.small_pool.freed‚ 5013835972096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3994.656516457 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37201588 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033426944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288173568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144€ý­å2Ö- _wandbruntime‚3995  _runtime‚3995¤šiýê&×äM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78434433657488e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18337743 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709777247641600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046053888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43507893 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18337480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162462561529856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10100637 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43497902 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717113 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4962046053888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43497902 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714732056632822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43507893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4954808991222 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441220 _runtime‚3995.268615372 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37209218 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 _step‚385 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37209218 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8236846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5014867144704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚4699XÕ´ï&ØéM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18337743 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709777247641600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046053888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43507893 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18337480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162462561529856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10100637 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43497902 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717113 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4962046053888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43497902 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714732056632822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43507893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4954808991222 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441220 _runtime‚3995.268615372 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37209218 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 _step‚385 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37209218 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8236846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5014867144704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚4699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78434433657488e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072àQ ^2Ù- _wandbruntime‚3995  _runtime‚3995ŠùêÚ‚ trainer/global_step‚4699 $ _timestamp‚1.784344336578898e+09 0loss/FRt1/assignment_loss‚1.7409950214599774 2loss/SRqqt1/assignment_loss‚0.5537766107347779 2loss/FBt1/assignment_loss‚0.013113307522017153 /loss/FRt1/detection_loss‚0.5693808898037358 0loss/FRt2/assignment_loss‚0.9101918067417892 1loss/FBt2/assignment_loss‚0.01817945010320793 2loss/SRqqt2/assignment_loss‚0.5215737863311444 /loss/FRt2/detection_loss‚1.2653364094317772 1loss/SRqqt1/detection_loss‚1.0948625409518908 1loss/SRqqt2/detection_loss‚0.6300446733039744 %loss/total_loss‚8.684916464398961 _runtime‚3995.272216292 _step‚386 /loss/FBt1/detection_loss‚0.8859073698263744 /loss/FBt2/detection_loss‚0.4815545981882938 epoch‚11Ÿ¡ú Û‡ /loss/FRt2/detection_loss‚1.2653364094317772 1loss/SRqqt1/detection_loss‚1.0948625409518908 1loss/SRqqt2/detection_loss‚0.6300446733039744 %loss/total_loss‚8.684916464398961 _runtime‚3995.272216292 _step‚386 /loss/FBt1/detection_loss‚0.8859073698263744 /loss/FBt2/detection_loss‚0.4815545981882938 epoch‚11 trainer/global_step‚4699 $ _timestamp‚1.784344336578898e+09 0loss/FRt1/assignment_loss‚1.7409950214599774 2loss/SRqqt1/assignment_loss‚0.5537766107347779 2loss/FBt1/assignment_loss‚0.013113307522017153 /loss/FRt1/detection_loss‚0.5693808898037358 0loss/FRt2/assignment_loss‚0.9101918067417892 1loss/FBt2/assignment_loss‚0.01817945010320793 2loss/SRqqt2/assignment_loss‚0.5215737863311444‚öÍ7“Ü‚(:ˆ šÞëÒ”Ëï” gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÞmÇÝ‚(:¼ šÞëÒßâö– disk.sda3.in‚0 disk.rbd1.out‚ 10.08203125$ memory_percent‚96.93217277526855 network.sent‚ 7.327163e+06 disk.sda3.out‚ 123.7421875 disk.sda1.in‚0' disk.nvme0c0n1.out‚13708.8154296875 disk.nvme0n1.in‚ 8356.671875' proc.memory.availableMB‚ 502.6328125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚13708.8154296875" proc.memory.rssMB‚ 2720.6171875 disk.sda1.out‚0 disk.dm-0.in‚ 8356.671875% disk.nvme0n1.out‚13708.8154296875 network.recv‚ 1.094643e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 123.7421875 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚596.7004241943359! disk.nvme0c0n1.in‚ 8356.671875 disk.sda.out‚ 123.7421875 disk.sda2.in‚0* proc.memory.percent‚16.605329513549805 cpu‚26.05394305391996) disk./.usagePercent‚40.04506613268547 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda.in‚0¾E”/Þ‚(:’ ©ÞëÒÎ…á” gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.162' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.83239999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¿¨Çß‚(:¼ ©ÞëÒäìÀ–% disk.nvme0n1.out‚13711.9970703125* proc.memory.percent‚16.605329513549805 cpu‚26.052699255699597 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 8469.42578125 disk.rbd1.in‚0$ disk./.usageGB‚596.7004814147949 disk.dm-1.in‚0 disk.sda.out‚ 124.03125# memory_percent‚96.9301700592041 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 124.03125 disk.sda2.out‚0 disk.rbd1.out‚ 10.14453125 proc.cpu.threads‚17) disk./.usagePercent‚40.04506997279848 disk.dm-1.out‚ 124.03125' proc.memory.availableMB‚ 502.9609375 network.sent‚ 7.390164e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 network.recv‚ 1.100289e+06 disk.sda.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2720.6171875' disk.nvme0c0n1.out‚13711.9970703125 disk.dm-0.in‚ 8469.42578125" disk.dm-0.out‚13711.9970703125! disk.nvme0n1.in‚ 8469.42578125"CCa2à- _wandbruntime‚4034  _runtime‚4034®ÿ4÷ÆáÀ & lr-AdamW/pg1‚0.0003966964812592565 & lr-AdamW/pg2‚0.0003966964812592565 trainer/global_step‚4749 $ _timestamp‚1.784344375974232e+09 _runtime‚4034.667835894 _step‚387Í ÄÂËâÅ $ _timestamp‚1.784344375974232e+09 _runtime‚4034.667835894 _step‚387 & lr-AdamW/pg1‚0.0003966964812592565 & lr-AdamW/pg2‚0.0003966964812592565 trainer/global_step‚4749ƒV4Ú+2ã- _wandbruntime‚4034  _runtime‚4034*pÀçäÜO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035156992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18523614 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164150798123520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37583188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43953279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5004073874406 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728928768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070267904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159085614983168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5065100854784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚725667676778496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5065183140352 \ODeviceStatsMonitor.]öpû on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676778496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720655964889088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8322351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843443759753695e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035156992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164150697225728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18523862 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚721960684987624 trainer/global_step‚4749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070267904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0[?ç'åáO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070267904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159085614983168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5065100854784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424296 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚725667676778496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5065183140352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676778496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720655964889088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8322351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843443759753695e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035156992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164150697225728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18523862 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚721960684987624 trainer/global_step‚4749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070267904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035156992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18523614 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164150798123520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37583188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43953279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5004073874406 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728928768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169„k¡°Å“æ‚(:ˆ ¸ÞëÒê´¸“ gpu.0.gpu‚71.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–i¥–Òç‚(:Ç ¸ÞëÒšºù– proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 10.14453125 network.sent‚ 7.395139e+06 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 8470.2578125 disk.dm-0.in‚ 8470.2578125$ memory_percent‚96.93009853363037 disk.sda1.in‚0( disk.nvme0c0n1.out‚13722.40087890625 disk.sda2.in‚0 disk.nvme0n1.in‚ 8470.2578125 disk.rbd0.in‚0 disk.sda.out‚ 124.16796875& disk.nvme0n1.out‚13722.40087890625 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚13722.40087890625 disk.sda.in‚0" proc.memory.rssMB‚ 2720.6171875 cpu‚26.051452968260424 network.recv‚ 1.10347e+06 disk.dm-1.out‚ 124.16796875 disk.sda2.out‚0* disk./.usagePercent‚40.045068180745744$ disk./.usageGB‚596.7004547119141 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 124.16796875( proc.memory.availableMB‚ 502.97265625* proc.memory.percent‚16.605329513549805ì%ÄÒ2è- _wandbruntime‚4035  _runtime‚4035HC?ì&éæM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18527401 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚4035.282802611 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7843443765890498e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553165 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43952354 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741555712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37590818 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10203308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543172 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720803205918720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164185253029888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5066116524032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543172 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37590818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10203572 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720803205918720 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962345 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚720803188879360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930435072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722108206713330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164185141653504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082902016 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962345 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082902016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18527401 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚4035.282802611 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7843443765890498e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553165 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43952354 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741555712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37590818 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10203308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543172 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720803205918720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164185253029888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5066116524032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543172 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37590818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10203572 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720803205918720 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962345 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚720803188879360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930435072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722108206713330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164185141653504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082902016 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962345 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082902016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 BÆg¿ð‚(:’ ÖÞëÒ¡þË“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r bÐñ‚(:Å ÖÞëÒê·ì•& disk.nvme0n1.out‚13734.68115234375 disk.sda3.out‚ 125.08984375 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 8471.22265625 disk.sda2.out‚0 disk.rbd1.out‚ 10.20703125) disk./.usagePercent‚40.04508046910738 disk.sda2.in‚0! disk.nvme0n1.in‚ 8471.22265625$ memory_percent‚96.93222045898438 network.recv‚ 1.113369e+06 disk.dm-1.out‚ 125.08984375 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8471.22265625( disk.nvme0c0n1.out‚13734.68115234375# disk.dm-0.out‚13734.68115234375* proc.memory.percent‚16.605329513549805 network.sent‚ 7.499456e+06 disk.sda1.out‚0 disk.rbd1.in‚0# proc.memory.availableMB‚502.625" proc.memory.rssMB‚ 2720.6171875 cpu‚26.049136339462507 disk.sda.out‚ 125.08984375$ disk./.usageGB‚596.7006378173828 disk.drbd2165.out‚0 disk.sda.in‚0‡¨J2ò- _wandbruntime‚4074  _runtime‚4074˜ÐbÉóà ' lr-AdamW/pg1‚0.00039553571051012326 ' lr-AdamW/pg2‚0.00039553571051012326 trainer/global_step‚4799 % _timestamp‚1.7843444159605393e+09 _runtime‚4074.654094378 _step‚391Ñv*ÎôÈ trainer/global_step‚4799 % _timestamp‚1.7843444159605393e+09 _runtime‚4074.654094378 _step‚391 ' lr-AdamW/pg1‚0.00039553571051012326 ' lr-AdamW/pg2‚0.00039553571051012326‡|ÇÂP2õ- _wandbruntime‚4074  _runtime‚4074M«r¶â'öÜO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362540 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782140416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397754 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397754 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37964786 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚4074.655465009 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37964788 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37964786 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18713431 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18713170 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160757052909568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765769728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚733080423761920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5062765769728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423761920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424430592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚728017999331328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10303830 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478882 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438÷ò«Sç'÷áO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765769728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚733080423761920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5062765769728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423761920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424430592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚728017999331328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10303830 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478882 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362540 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782140416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397754 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397754 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37964786 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚4074.655465009 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37964788 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37964786 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18713431 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18713170 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160757052909568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976ˆ.8î<2ø- _wandbruntime‚4075  _runtime‚4075SÏ ëùæM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5063778403840 % _timestamp‚1.7843444165745902e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437057536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389217 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117310810112 _runtime‚4075.268346135 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚733229035804160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677418496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416797 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165907792478208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10305533 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716876 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8411346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚728165240360960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406806 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907691587584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056041965038 ^nÜ ìJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18717138 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117228531712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10305792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169f-¿¡ñ&úëM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10305792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5063778403840 % _timestamp‚1.7843444165745902e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437057536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389217 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117310810112 _runtime‚4075.268346135 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚733229035804160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677418496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416797 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165907792478208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10305533 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716876 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8411346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚728165240360960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406806 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907691587584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056041965038 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18717138 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117228531712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0‰ZOwv2û- _wandbruntime‚4075  _runtime‚4075WŒ¤¯üü 0loss/SRqqt1/detection_loss‚1.004080596344935 0loss/SRqqt2/detection_loss‚0.675601373004487 /loss/FBt2/detection_loss‚0.5492680458368601 trainer/global_step‚4799 _runtime‚4075.271235296 epoch‚11 0loss/FRt2/assignment_loss‚1.0473238925706772 2loss/FBt2/assignment_loss‚0.003009381986644907 /loss/FRt1/detection_loss‚0.5849740609884604 #loss/total_loss‚8.7703967232823 /loss/FRt1/assignment_loss‚1.667886794499278 1loss/SRqqt1/assignment_loss‚0.586314936874973 2loss/SRqqt2/assignment_loss‚0.5734800279742583 /loss/FRt2/detection_loss‚1.2777912610713835 /loss/FBt1/detection_loss‚0.7006639160588384 $ _timestamp‚1.784344416577754e+09 _step‚394 1loss/FBt1/assignment_loss‚0.10000243607150282¿XUÚý /loss/FRt1/assignment_loss‚1.667886794499278 1loss/SRqqt1/assignment_loss‚0.586314936874973 2loss/SRqqt2/assignment_loss‚0.5734800279742583 /loss/FRt2/detection_loss‚1.2777912610713835 /loss/FBt1/detection_loss‚0.7006639160588384 $ _timestamp‚1.784344416577754e+09 _step‚394 1loss/FBt1/assignment_loss‚0.10000243607150282 0loss/SRqqt1/detection_loss‚1.004080596344935 0loss/SRqqt2/detection_loss‚0.675601373004487 /loss/FBt2/detection_loss‚0.5492680458368601 trainer/global_step‚4799 _runtime‚4075.271235296 epoch‚11 0loss/FRt2/assignment_loss‚1.0473238925706772 2loss/FBt2/assignment_loss‚0.003009381986644907 /loss/FRt1/detection_loss‚0.5849740609884604 #loss/total_loss‚8.7703967232823Štd‘þ‚(:† åÞëÒÆÿ·“ gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uúÂ)Çÿ‚(:¼ åÞëÒÓñ‚— disk.drbd2165.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚13741.759765625* proc.memory.percent‚16.605329513549805 cpu‚26.052531382024448 network.recv‚ 1.11723e+06) disk./.usagePercent‚40.04511810221492 disk.sda2.in‚0 disk.sda1.in‚0! disk.dm-0.out‚13741.759765625 network.sent‚ 7.504545e+06$ disk./.usageGB‚596.7011985778809 disk.rbd1.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2720.6171875 disk.sda3.in‚0 disk.sda.out‚ 125.42578125$ proc.memory.availableMB‚502.3125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 125.42578125 disk.dm-0.in‚ 8472.0546875 disk.sda3.out‚ 125.42578125$ memory_percent‚96.93412780761719 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 8472.0546875 disk.sda.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 8472.0546875$ disk.nvme0n1.out‚13741.759765625 disk.rbd1.out‚ 10.26953125 disk.drbd2165.in‚0«kj”2€- _wandbruntime‚4092  _runtime‚4092Æ;®;“{ 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8534770011901855 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45200467109680176 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20718799531459808 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5950413942337036 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8437453508377075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7690814137458801 6validation_average_jet_accuracy‚0.3812771439552307 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9701343774795532 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02417716570198536 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.691948652267456 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8129295110702515 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021177388727664948 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7858951091766357 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30152735114097595 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8369237780570984 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4823194146156311 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25605347752571106 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07378120720386505 % _timestamp‚1.7843444337173913e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6305040717124939 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.743994951248169 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7159703373908997 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9883524775505066 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5633122324943542 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18221166729927063 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 epoch‚11 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992695689201355 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6700112819671631 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7683273553848267 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982032775878906 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03841441124677658 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9849426746368408 )loss/val_total_loss‚8.291271812563508 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.090770423412323 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7303631901741028 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6516510248184204 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47182291746139526 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19345933198928833 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6599757671356201 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7581042647361755 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8767850995063782 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768073320388794 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951313734054565 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2430018186569214 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6386191844940186 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2444818764925003 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7706810832023621 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6135092377662659 *jet/accuracy_3_of_4‚0.0955251157283783 .particle/accuracy_4_of_4‚0.929198145866394 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3977334499359131 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46515345573425293 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19519107043743134 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5520833730697632 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4574279189109802 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916665732860565 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3032965362071991 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955604076385498 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.193757563829422 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6732298731803894 _step‚395 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9395329356193542 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7120705842971802 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24447831511497498 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7662542462348938 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8767850995063782 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9986832737922668 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04593561589717865 *jet/accuracy_2_of_4‚0.8150331377983093 3 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987548828125 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6657517552375793 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6732298731803894 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9127290844917297 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1997414082288742 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7558528184890747 *jet/accuracy_1_of_3‚0.9132115244865417 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3367573916912079 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02531115710735321 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945448637008667 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25522804260253906 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021177388727664948 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6696733832359314 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2444818764925003 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09200631827116013 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031045308336615562 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6735978126525879 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6245001554489136 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4818418622016907 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050891391932964325 jet/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.669521689414978 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7703728675842285 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_2‚0.1524468958377838 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9137406349182129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7145108580589294 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6762245893478394 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026946403086185455 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987633228302002 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864048957824707 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7192931175231934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5595259070396423 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33419710397720337 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7437624931335449 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23594024777412415 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1959882229566574 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39990538358688354 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22575417160987854 +jet/accuracy_1_of_1‚0.34255605936050415 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8647795915603638 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +jet/accuracy_3_of_3‚0.07594481110572815 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.890312910079956 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8516245484352112 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9850670695304871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18571630120277405 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7782124280929565 /particle/accuracy_3_of_4‚0.9910430908203125 jet/accuracy_5_of_5‚0 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85496985912323 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.85496985912323 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 &particle/f_score‚0.643828809261322 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20674780011177063 particle/accuracy_1_of_4‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05931772664189339 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2551969587802887 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20761390030384064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7634890675544739 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8138560056686401 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2033553123474121 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6875611543655396 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10959895700216293 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.634114146232605 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7362504601478577 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2138570100069046 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.752783477306366 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939673542976379 &êËÉ‹z Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02483302354812622 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02409171685576439 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 trainer/global_step‚4799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599355697631836 *jet/accuracy_2_of_3‚0.5756256580352783 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚4092.411708669 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534Ê<ž=‚˜{ 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4574279189109802 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916665732860565 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3032965362071991 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955604076385498 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.193757563829422 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6732298731803894 _step‚395 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9395329356193542 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7120705842971802 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24447831511497498 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7662542462348938 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8767850995063782 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9986832737922668 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04593561589717865 *jet/accuracy_2_of_4‚0.8150331377983093 3 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987548828125 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6657517552375793 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6732298731803894 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9127290844917297 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1997414082288742 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7558528184890747 *jet/accuracy_1_of_3‚0.9132115244865417 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3367573916912079 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02531115710735321 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945448637008667 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25522804260253906 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021177388727664948 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6696733832359314 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2444818764925003 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09200631827116013 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031045308336615562 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6735978126525879 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6245001554489136 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4818418622016907 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050891391932964325 jet/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.669521689414978 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7703728675842285 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_2‚0.1524468958377838 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9137406349182129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7145108580589294 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6762245893478394 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026946403086185455 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987633228302002 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864048957824707 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7192931175231934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5595259070396423 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33419710397720337 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7437624931335449 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23594024777412415 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1959882229566574 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39990538358688354 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22575417160987854 +jet/accuracy_1_of_1‚0.34255605936050415 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8647795915603638 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +jet/accuracy_3_of_3‚0.07594481110572815 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.890312910079956 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8516245484352112 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9850670695304871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18571630120277405 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7782124280929565 /particle/accuracy_3_of_4‚0.9910430908203125 jet/accuracy_5_of_5‚0 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.85496985912323 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.85496985912323 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 &particle/f_score‚0.643828809261322 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20674780011177063 particle/accuracy_1_of_4‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05931772664189339 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2551969587802887 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20761390030384064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7634890675544739 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8138560056686401 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2033553123474121 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6875611543655396 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10959895700216293 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.634114146232605 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7362504601478577 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2138570100069046 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.752783477306366 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939673542976379 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02483302354812622 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02409171685576439 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 trainer/global_step‚4799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599355697631836 *jet/accuracy_2_of_3‚0.5756256580352783 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚4092.411708669 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8534770011901855 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45200467109680176 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20718799531459808 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5950413942337036 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8437453508377075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7690814137458801 6validation_average_jet_accuracy‚0.3812771439552307 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9701343774795532 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02417716570198536 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.691948652267456 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8129295110702515 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021177388727664948 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7858951091766357 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30152735114097595 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8369237780570984 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4823194146156311 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25605347752571106 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07378120720386505 % _timestamp‚1.7843444337173913e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6305040717124939 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.743994951248169 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7159703373908997 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9883524775505066 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5633122324943542 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18221166729927063 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 epoch‚11 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992695689201355 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6700112819671631 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7683273553848267 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982032775878906 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03841441124677658 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9849426746368408 )loss/val_total_loss‚8.291271812563508 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.090770423412323 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7303631901741028 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6516510248184204 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47182291746139526 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19345933198928833 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6599757671356201 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7581042647361755 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8767850995063782 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768073320388794 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951313734054565 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2430018186569214 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6386191844940186 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2444818764925003 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7706810832023621 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6135092377662659 *jet/accuracy_3_of_4‚0.0955251157283783 .particle/accuracy_4_of_4‚0.929198145866394 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3977334499359131 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46515345573425293 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19519107043743134 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5520833730697632‹«=7‘ƒ‚(:† ôÞëÒ¿ÿ›” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚216.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.2584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½-wéÉ„‚(:¾ ôÞëÒØ¡•– network.recv‚ 1.122644e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 125.68359375# disk.nvme0c0n1.in‚ 8606.38671875" proc.memory.rssMB‚ 2720.7265625) proc.memory.percent‚16.60599708557129 network.sent‚ 7.567357e+06 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 8606.38671875) disk./.usagePercent‚40.04519951261081 disk.dm-0.in‚ 8606.38671875 disk.sda.out‚ 125.68359375 disk.sda3.in‚0& proc.memory.availableMB‚ 743.140625 cpu‚26.061990703443005$ disk./.usageGB‚596.7024116516113 disk.rbd0.out‚0 disk.dm-0.out‚13996.12890625$ memory_percent‚95.46422958374023 disk.rbd1.out‚ 10.26953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 125.68359375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚13996.12890625 disk.rbd1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚13996.12890625$&¦…‚(:„ ƒßëÒšû™” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øe+ц‚(:Æ ƒßëÒ¿•Î• cpu‚26.060746518291356 disk.sda.out‚ 125.86328125( disk./.usagePercent‚40.0450925014615 disk.dm-0.in‚ 8606.38671875 disk.sda2.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚14041.74658203125# disk.dm-0.out‚14041.74658203125& disk.nvme0n1.out‚14041.74658203125 disk.sda3.out‚ 125.86328125 disk.rbd1.out‚ 10.33203125 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 743.1953125 network.recv‚ 1.127994e+06 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 7.62469e+06# disk.nvme0c0n1.in‚ 8606.38671875 disk.sda3.in‚0$ memory_percent‚95.46389579772949 disk.dm-1.out‚ 125.86328125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 8606.38671875 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.7265625) proc.memory.percent‚16.60599708557129$ disk./.usageGB‚596.7008171081543 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0(¡I哇‚(:ˆ ’ßëÒΉ¥” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02r»Ëˆ‚(:À ’ßëÒ€„×– disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8607.21875 disk.sda1.in‚0 disk.dm-1.out‚ 126.19921875# disk.dm-0.out‚14261.01611328125 disk.sda.in‚0( proc.memory.availableMB‚ 743.30078125" proc.memory.rssMB‚ 2720.7265625* disk./.usagePercent‚40.045171095774506 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 8607.21875 disk.nvme0n1.in‚ 8607.21875 cpu‚26.059369613131896$ disk./.usageGB‚596.7019882202148 disk.drbd2165.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚14261.01611328125 disk.sda3.in‚0 disk.sda3.out‚ 126.19921875$ memory_percent‚95.46325206756592 network.sent‚ 7.62956e+06 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.33203125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚14261.01611328125 disk.sda.out‚ 126.19921875) proc.memory.percent‚16.60599708557129 network.recv‚ 1.131274e+06 disk.sda2.out‚0̆…2‰- _wandbruntime‚4133  _runtime‚41332²“½ÇŠÁ & lr-AdamW/pg2‚0.0003943639569278518 trainer/global_step‚4849 % _timestamp‚1.7843444752695534e+09 _runtime‚4133.963118452 _step‚396 & lr-AdamW/pg1‚0.0003943639569278518ÝçÃóÌ‹Æ & lr-AdamW/pg1‚0.0003943639569278518 & lr-AdamW/pg2‚0.0003943639569278518 trainer/global_step‚4849 % _timestamp‚1.7843444752695534e+09 _runtime‚4133.963118452 _step‚396ŒœU-Z2Œ- _wandbruntime‚4133  _runtime‚4133¨ â'ÜO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087954 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733516000964608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470603776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8499371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44922533 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18939617 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185652341760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742440828982272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162444384323584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843444752707527e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737309634502656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123346 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389585 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977¹,b•ŽáO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44932510 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167630036665344 _runtime‚4133.964466294 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185652341760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742440828982272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162444384323584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843444752707527e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737309634502656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123346 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389585 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087954 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733516000964608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470603776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8499371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44922533 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18939617 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162477813082112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733662511636480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167664499184640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5132207113728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5124378177180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5131865767424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733662494597120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10442579 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4134.577229838 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162477831694336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5131865767424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18943695 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18943437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742588724260352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931585 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738786532224298 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207113728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167664597978112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742589082646016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269vÌqîð&‘êM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5131865767424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18943695 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18943437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742588724260352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931585 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738786532224298 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207113728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167664597978112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742589082646016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162477813082112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733662511636480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167664499184640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5132207113728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5124378177180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5131865767424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733662494597120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10442579 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4134.577229838 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162477831694336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ž'9g‰2’- _wandbruntime‚4134  _runtime‚4134> "²“‚ 1loss/FBt1/assignment_loss‚0.06577795968419614 1loss/SRqqt1/detection_loss‚0.9418177323200488 %loss/total_loss‚8.835277443465426 trainer/global_step‚4849 epoch‚12 0loss/FRt1/assignment_loss‚1.6879010152914384 2loss/SRqqt1/assignment_loss‚0.5635215746638318 1loss/SRqqt2/detection_loss‚0.7057827430472681 /loss/FBt1/detection_loss‚1.2785325705220825 /loss/FBt2/detection_loss‚0.4141418616840146 3loss/SRqqt2/assignment_loss‚0.48141573009012695 2loss/FBt2/assignment_loss‚0.008848998400743708 $ _timestamp‚1.784344475887109e+09 _runtime‚4134.580405274 /loss/FRt2/assignment_loss‚0.871945242250235 /loss/FRt1/detection_loss‚0.6055329339930984 /loss/FRt2/detection_loss‚1.2100590815183419 _step‚399FSƤ ”‡ epoch‚12 0loss/FRt1/assignment_loss‚1.6879010152914384 2loss/SRqqt1/assignment_loss‚0.5635215746638318 1loss/SRqqt2/detection_loss‚0.7057827430472681 /loss/FBt1/detection_loss‚1.2785325705220825 /loss/FBt2/detection_loss‚0.4141418616840146 3loss/SRqqt2/assignment_loss‚0.48141573009012695 2loss/FBt2/assignment_loss‚0.008848998400743708 $ _timestamp‚1.784344475887109e+09 _runtime‚4134.580405274 /loss/FRt2/assignment_loss‚0.871945242250235 /loss/FRt1/detection_loss‚0.6055329339930984 /loss/FRt2/detection_loss‚1.2100590815183419 _step‚399 1loss/FBt1/assignment_loss‚0.06577795968419614 1loss/SRqqt1/detection_loss‚0.9418177323200488 %loss/total_loss‚8.835277443465426 trainer/global_step‚4849DØŒ“•‚(:ˆ ¡ßëÒª²Á” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅV+oÁ–‚(:¶ ¡ßëÒÿ¤ò–* disk./.usagePercent‚40.045108885943684$ disk./.usageGB‚596.7010612487793 disk.sda2.out‚0 disk.nvme0n1.in‚ 8607.21875" disk.dm-0.out‚14276.8115234375 disk.sda2.in‚0% disk.nvme0n1.out‚14276.8115234375 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 126.578125' disk.nvme0c0n1.out‚14276.8115234375 disk.rbd1.in‚0 disk.sda1.in‚0# memory_percent‚95.4648494720459 disk.sda.out‚ 126.578125 disk.rbd1.out‚ 10.39453125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 8607.21875) proc.memory.percent‚16.60599708557129 cpu‚26.058133745949522 network.recv‚ 1.134881e+06 disk.sda3.in‚0 disk.sda3.out‚ 126.578125 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 7.647544e+06' proc.memory.availableMB‚ 743.0390625" proc.memory.rssMB‚ 2720.7265625 disk.sda.in‚0 disk.nvme0c0n1.in‚ 8607.21875 disk.drbd2165.out‚0Ý& “—‚(:ˆ °ßëÒÚ³š” gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚214.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.8822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñù&˜‚(:¿ °ßëÒ˜æé•& proc.memory.availableMB‚ 743.015625$ memory_percent‚95.46499252319336$ disk./.usageGB‚596.7011756896973) proc.memory.percent‚16.60599708557129 cpu‚26.0569232707774) disk./.usagePercent‚40.04511656616971 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8691.08203125 proc.cpu.threads‚17 disk.dm-1.in‚0( disk.nvme0c0n1.out‚14292.31884765625& disk.nvme0n1.out‚14292.31884765625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 10.39453125 network.sent‚ 7.697429e+06 diíùW‰¤sk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 126.703125 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 1.139803e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 126.703125# disk.dm-0.out‚14292.31884765625 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda3.out‚ 126.703125 disk.sda1.in‚0 disk.dm-0.in‚ 8691.08203125! disk.nvme0n1.in‚ 8691.08203125Xk›™‚(: ¿ßëÒ¨‹µ” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.224' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.84479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<*‰Îš‚(:à ¿ßëÒ£Üò” disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 7.702493e+06 disk.drbd2165.out‚0& disk.nvme0n1.out‚14299.43701171875 disk.sda3.out‚ 127.2890625 network.recv‚ 1.143587e+06 disk.dm-0.in‚ 8691.9140625 cpu‚26.055719837353397 disk.rbd0.out‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚14299.43701171875) proc.memory.percent‚16.60599708557129$ disk./.usageGB‚596.7011032104492 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 8691.9140625 disk.sda3.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 127.2890625 disk.sda.out‚ 127.2890625$ memory_percent‚95.46372890472412" proc.memory.rssMB‚ 2720.7265625# disk.dm-0.out‚14299.43701171875 disk.nvme0n1.in‚ 8691.9140625 disk.rbd1.out‚ 10.39453125( proc.memory.availableMB‚ 743.22265625) disk./.usagePercent‚40.04511170202657 disk.rbd0.in‚0 disk.drbd2165.in‚0¥‰m®2›- _wandbruntime‚4173  _runtime‚4173ûSùãÇœÁ _step‚400 & lr-AdamW/pg1‚0.0003931812957719851 & lr-AdamW/pg2‚0.0003931812957719851 trainer/global_step‚4899 % _timestamp‚1.7843445150528414e+09 _runtime‚4173.746420524Üi±AÌÆ & lr-AdamW/pg2‚0.0003931812957719851 trainer/global_step‚4899 % _timestamp‚1.7843445150528414e+09 _runtime‚4173.746420524 _step‚400 & lr-AdamW/pg1‚0.00039318129577198519Æq2ž- _wandbruntime‚4173  _runtime‚4173n¶ìâ'ŸÜO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8586621 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889981440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19131185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173643060364 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚749853217587200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548642304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181889981440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚746015160579212 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚749853575965696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164115822249984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10544284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217587200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169352877139968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164115840862208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 trainer/global_step‚4899 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19130902 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225637 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10544564 % _timestamp‚1.7843445150538878e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376985 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548642304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5236947700736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616ÙñŸËç' áO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19131185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173643060364 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚749853217587200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548642304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181889981440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚746015160579212 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚749853575965696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164115822249984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10544284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217587200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169352877139968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164115840862208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 trainer/global_step‚4899 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19130902 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225637 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10544564 % _timestamp‚1.7843445150538878e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376985 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548642304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5236947700736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8586621 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889981440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438‘*íÚÇ2¡- _wandbruntime‚4174  _runtime‚4174 ‰s ´&¢æM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8588366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164149269620736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744818909974528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164149251008512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902615552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19135002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10546636 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242321 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38856286 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386037 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750001829629440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45386037 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5182902615552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5238056347136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252314 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚4174.360051056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169387222980096 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252314 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5237971971584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843445156663296e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5174994664088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚750001471243776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561269248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396028 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚4899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10546383 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829629440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19134746 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252314 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚4174.360051056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169387222980096 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252314 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5237971971584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843445156663296e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5174994664088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚750001471243776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561269248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396028 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚4899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10546383 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829629440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19134746 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744818909974528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164149251008512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902615552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19135002’¶tq2¤- _wandbruntime‚4174  _runtime‚4174&æŒÅ¥€ 0loss/FRt1/assignment_loss‚1.7469885127444098 0loss/FRt2/assignment_loss‚0.9302781062449762 2loss/SRqqt1/assignment_loss‚0.5755645070849967 1loss/SRqqt2/assignment_loss‚0.515791936567904 1loss/FBt2/assignment_loss‚0.06919314412665432 .loss/FRt1/detection_loss‚0.568495333299253 /loss/FRt2/detection_loss‚1.3188531201050715 1loss/SRqqt2/detection_loss‚0.6512462944190398 0loss/SRqqt1/detection_loss‚0.974821066212628 epoch‚12 trainer/global_step‚4899 % _timestamp‚1.7843445156690729e+09 _step‚403 /loss/FBt1/detection_loss‚0.7875947173541555 %loss/total_loss‚8.618281170343408 2loss/FBt1/assignment_loss‚0.008773820493821369 /loss/FBt2/detection_loss‚0.4706806116904993 _runtime‚4174.362386987cj ¦… 1loss/FBt2/assignment_loss‚0.06919314412665432 .loss/FRt1/detection_loss‚0.568495333299253 /loss/FRt2/detection_loss‚1.3188531201050715 1loss/SRqqt2/detection_loss‚0.6512462944190398 0loss/SRqqt1/detection_loss‚0.974821066212628 epoch‚12 trainer/global_step‚4899 % _timestamp‚1.7843445156690729e+09 _step‚403 /loss/FBt1/detection_loss‚0.7875947173541555 %loss/total_loss‚8.618281170343408 2loss/FBt1/assignment_loss‚0.008773820493821369 /loss/FBt2/detection_loss‚0.4706806116904993 _runtime‚4174.362386987 0loss/FRt1/assignment_loss‚1.7469885127444098 0loss/FRt2/assignment_loss‚0.9302781062449762 2loss/SRqqt1/assignment_loss‚0.5755645070849967 1loss/SRqqt2/assignment_loss‚0.515791936567904“ãc#מ§‚(:“ ÎßëÒË÷—“ gpu.0.gpu‚84.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÖSçϨ‚(:Ä ÎßëÒÂî× network.sent‚ 7.720473e+06 network.recv‚ 1.146889e+06 disk.nvme0n1.in‚ 8691.9140625 disk.rbd1.in‚0 disk.rbd1.out‚ 10.45703125) proc.memory.percent‚16.60599708557129& disk.nvme0n1.out‚14305.55322265625 disk.sda.out‚ 128.46484375 cpu‚26.054538272709024 disk.sda2.out‚0 disk.dm-1.out‚ 128.46484375 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 8691.9140625# disk.dm-0.out‚14305.55322265625 disk.rbd0.out‚0) disk./.usagePercent‚40.04510683788342 disk.dm-1.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda1.out‚0( disk.nvme0c0n1.out‚14305.55322265625& proc.memory.availableMB‚ 742.734375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0$ memory_percent‚95.46670913696289$ disk./.usageGB‚596.7010307312012 disk.sda3.out‚ 128.46484375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 8691.9140625§YÄ“©‚(:ˆ ÝßëÒ±„ñ“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶^Д˪‚(:À ÝßëÒ”¤Î– disk.dm-1.out‚ 130.53125 network.recv‚ 1.152379e+06) disk./.usagePercent‚40.04490920006698 disk.sda.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda3.out‚ 130.53125 disk.sda.out‚ 130.53125& disk.nvme0n1.out‚14321.00927734375 disk.sda3.in‚0# disk.dm-0.out‚14321.00927734375 cpu‚26.053233243660905 proc.cpu.threads‚17( proc.memory.availableMB‚ 742.77734375 disk.sda2.in‚0 network.sent‚ 7.770456e+06! disk.nvme0n1.in‚ 8692.87890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.46644687652588 disk.dm-1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 8692.87890625( disk.nvme0c0n1.out‚14321.00927734375) proc.memory.percent‚16.60599708557129$ disk./.usageGB‚596.6980857849121 disk.drbd2165.in‚0 disk.dm-0.in‚ 8692.87890625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.45703125:Z…G2«- _wandbruntime‚4213  _runtime‚4213Ug³É¬Ã trainer/global_step‚4949 % _timestamp‚1.7843445550493999e+09 _runtime‚4213.742988362 _step‚404 ' lr-AdamW/pg1‚0.00039198780300263926 ' lr-AdamW/pg2‚0.00039198780300263926+»ÐÏÎ­È ' lr-AdamW/pg1‚0.00039198780300263926 ' lr-AdamW/pg2‚0.00039198780300263926 trainer/global_step‚4949 % _timestamp‚1.7843445550493999e+09 _runtime‚4213.742988362 _step‚404”¦Ã.˜2®- _wandbruntime‚4213  _runtime‚4213Ô ç€á'¯ÛO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171075558025216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964570624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5288188562432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097162 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45841414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752033720426496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165787278788608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322949120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10647896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5232244144128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39230252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075448738816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752033737465856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585483264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39230252 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831437 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964570624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322949120±Ä°àO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061689 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5232585483264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071668 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 $ _timestamp‚1.784344555050501e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097162 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45841414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752033720426496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165787278788608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322949120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10647896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5232244144128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39230252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075448738816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752033737465856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585483264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39230252 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831437 YEDeviceStats†Î\fb Monitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964570624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322949120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171075558025216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964570624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5288188562432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600994 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19325589 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45840489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598117376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748313561784320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256771072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088366 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078373 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171110012272128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088366 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39237884 _step‚406 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078373 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚752180961456128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165820688934912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10649976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256771072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616/‹4úñ&³ëM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45840489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598117376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748313561784320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256771072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088366 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078373 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171110012272128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088366 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39237884 _step‚406 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078373 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚752180961456128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165820688934912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10649976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚4949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256771072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171109907187200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611150996 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600994 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19325589 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888–’—2´- _wandbruntime‚4214  _runtime‚4214jÔ7µ 1loss/SRqqt1/assignment_loss‚0.573015340539234 2loss/SRqqt2/assignment_loss‚0.5378266447971293 /loss/FBt1/detection_loss‚0.5455080334296716 0loss/FBt2/detection_loss‚0.41074442240441666 epoch‚12 % _timestamp‚1.7843445556672924e+09 _runtime‚4214.360601432 1loss/FBt1/assignment_loss‚0.03151591476968869 /loss/FRt2/detection_loss‚1.3442239812060603 1loss/SRqqt2/detection_loss‚0.6361206421990042 trainer/global_step‚4949 0loss/FRt2/assignment_loss‚0.8254362723701879 2loss/FBt2/assignment_loss‚0.010948213634394124 /loss/FRt1/detection_loss‚0.5488917420426926 1loss/SRqqt1/detection_loss‚0.9344433718899429 $loss/total_loss‚8.18514614412852 _step‚407 /loss/FRt1/assignment_loss‚1.786471564846098÷ö\¦ ¶† /loss/FRt2/detection_loss‚1.3442239812060603 1loss/SRqqt2/detection_loss‚0.6361206421990042 trainer/global_step‚4949 0loss/FRt2/assignment_loss‚0.8254362723701879 2loss/FBt2/assignment_loss‚0.010948213634394124 /loss/FRt1/detection_loss‚0.5488917420426926 1loss/SRqqt1/detection_loss‚0.9344433718899429 $loss/total_loss‚8.18514614412852 _step‚407 /loss/FRt1/assignment_loss‚1.786471564846098 1loss/SRqqt1/assignment_loss‚0.573015340539234 2loss/SRqqt2/assignment_loss‚0.5378266447971293 /loss/FBt1/detection_loss‚0.5455080334296716 0loss/FBt2/detection_loss‚0.41074442240441666 epoch‚12 % _timestamp‚1.7843445556672924e+09 _runtime‚4214.360601432 1loss/FBt1/assignment_loss‚0.03151591476968869—j0¿·‚(:’ ìßëÒç©¡“ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.162' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.83239999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E;»Ò¸‚(:Ç ìßëÒæ³ü•& disk.nvme0n1.out‚14333.70361328125 disk.sda.out‚ 131.12890625 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚95.46639919281006 network.recv‚ 1.155508e+06$ disk./.usageGB‚596.6980361938477 disk.dm-1.out‚ 131.12890625 disk.nvme0n1.in‚ 8821.6484375 network.sent‚ 7.775408e+06* disk./.usagePercent‚40.044905871969036 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 8821.6484375 disk.dm-0.in‚ 8821.6484375 disk.rbd1.in‚0 disk.rbd1.out‚ 10.58203125( proc.memory.availableMB‚ 742.78515625) proc.memory.percent‚16.60599708557129 cpu‚26.051992863019173 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 131.12890625" proc.memory.rssMB‚ 2720.7265625 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚14333.70361328125# disk.dm-0.out‚14333.70361328125 disk.sda.in‚0Jº“¹‚(:ˆ ûßëÒ‘§½” gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚147.604' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.5208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gʉù¿º‚(:´ ûßëÒ⤖ disk.rbd1.in‚0' proc.memory.availableMB‚ 742.6015625 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 14338.4140625 disk.dm-0.in‚ 8821.6484375$ disk./.usageGB‚596.6981048583984 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 8821.6484375" proc.memory.rssMB‚ 2720.7265625 disk.rbd0.out‚0 disk.dm-1.in‚0) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 network.sent‚ 7.83838e+06) disk./.usagePercent‚40.04491048010465 disk.rbd0.in‚0 disk.sda3.out‚ 131.4609375 cpu‚26.05086670246645$ memory_percent‚95.46751976013184 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 8821.6484375 disk.dm-0.out‚ 14338.4140625" disk.nvme0n1.out‚ 14338.4140625 disk.sda3.in‚0 network.recv‚ 1.161055e+06 disk.sda1.in‚0 disk.dm-1.out‚ 131.4609375 disk.sda.out‚ 131.4609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 10.58203125 disk.sda.in‚0Á¾êÒ‘»‚(:† ŠàëÒ´Àƒ” gpu.0.gpu‚23.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.057 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùOW^Ѽ‚(:Æ ŠàëÒòé‘—$ memory_percent‚95.46506404876709) proc.memory.percent‚16.60599708557129) disk./.usagePercent‚40.04481447727931 disk.sda.out‚ 132.05078125 disk.sda3.in‚0' disk.nvme0c0n1.out‚14350.5556640625" proc.memory.rssMB‚ 2720.7265625! disk.nvme0n1.in‚ 8822.61328125( proc.memory.availableMB‚ 743.00390625$ disk./.usageGB‚596.6966743469238 disk.dm-1.in‚0 disk.dm-1.out‚ 132.05078125# disk.nvme0c0n1.in‚ 8822.61328125 network.sent‚ 7.845063e+06 disk.sda2.out‚0 network.recv‚ 1.163631e+06 disk.sda.in‚0 disk.rbd1.out‚ 10.64453125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 8822.61328125 disk.sda2.in‚0% disk.nvme0n1.out‚14350.5556640625 disk.sda3.out‚ 132.05078125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 cpu‚26.049682760226993 proc.cpu.threads‚17" disk.dm-0.out‚14350.5556640625 disk.sda1.in‚0 disk.sda1.out‚0©<2½- _wandbruntime‚4253  _runtime‚4253@×ÄǾÁ trainer/global_step‚4999 % _timestamp‚1.7843445948626568e+09 _runtime‚4253.556316061 _step‚408 & lr-AdamW/pg1‚0.0003907835552756246 & lr-AdamW/pg2‚0.0003907835552756246£E Ì¿Æ _runtime‚4253.556316061 _step‚408 & lr-AdamW/pg1‚0.0003907835552756246 & lr-AdamW/pg2‚0.0003907835552756246 trainer/global_step‚4999 % _timestamp‚1.7843445948626568e+09˜cD ?2À- _wandbruntime‚4253  _runtime‚4253f»º²ÁÜO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚4253.557555812 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19512653 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10751535 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚759395771908096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚760767460740228 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755492601745408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocaÙ ¡0ted‚46295866 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939645952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚409 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711554048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069932544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5339445944832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329478 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285889 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4999 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843445948638153e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39611852 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711554048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172798162659840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897741 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5339355270656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5339355270656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚759395771908096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚760767460740228 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755492601745408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46295866 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939645952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚409 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711554048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069932544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5339445944832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329478 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285889 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4999 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843445948638153e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39611852 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584070 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711554048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172798162659840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0™'µ 2Ã- _wandbruntime‚4254  _runtime‚4254à õˆì&ÄæM KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952272896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5340467501056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465822 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759543029977088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167492145473536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764827323596288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293619200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39619486 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293619200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087902 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924418 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304932 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294941 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914425 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19516402 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952272896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764826965210624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10753539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294941 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843445954778259e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924418 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088T¨ÅëM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764827323596288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293619200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39619486 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293619200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087902 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924418 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304932 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294941 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914425 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19516402 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952272896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764826965210624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10753539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NdöBñIADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294941 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843445954778259e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924418 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952272896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5340467501056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465822 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759543029977088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167492145473536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0š»™Õ2Æ- _wandbruntime‚4254  _runtime‚4254›á¸Ç€ /loss/FRt2/assignment_loss‚0.871219487566697 3loss/SRqqt2/assignment_loss‚0.45538168719237876 .loss/FRt2/detection_loss‚1.277589419137298 /loss/FBt1/detection_loss‚1.1172404094422992 /loss/FBt2/detection_loss‚0.6293879426254823 _runtime‚4254.174372681 /loss/FRt1/detection_loss‚0.5948124176616001 1loss/SRqqt1/detection_loss‚0.9716884713533884 %loss/total_loss‚9.007530226303889 $ _timestamp‚1.784344595481083e+09 _step‚411 2loss/SRqqt1/assignment_loss‚0.5854534578672725 2loss/FBt2/assignment_loss‚0.019859253239969198 0loss/SRqqt2/detection_loss‚0.633220862560917 0loss/FRt1/assignment_loss‚1.7527562858414476 1loss/FBt1/assignment_loss‚0.09892053181513896 epoch‚12 trainer/global_step‚4999Ñ7„ È… 0loss/FRt1/assignment_loss‚1.7527562858414476 1loss/FBt1/assignment_loss‚0.09892053181513896 epoch‚12 trainer/global_step‚4999 /loss/FRt2/assignment_loss‚0.871219487566697 3loss/SRqqt2/assignment_loss‚0.45538168719237876 .loss/FRt2/detection_loss‚1.277589419137298 /loss/FBt1/detection_loss‚1.1172404094422992 /loss/FBt2/detection_loss‚0.6293879426254823 _runtime‚4254.174372681 /loss/FRt1/detection_loss‚0.5948124176616001 1loss/SRqqt1/detection_loss‚0.9716884713533884 %loss/total_loss‚9.007530226303889 $ _timestamp‚1.784344595481083e+09 _step‚411 2loss/SRqqt1/assignment_loss‚0.5854534578672725 2loss/FBt2/assignment_loss‚0.019859253239969198 0loss/SRqqt2/detection_loss‚0.633220862560917››ÓS“É‚(:ˆ ™àëÒ»Ô¢” gpu.0.gpu‚93.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BwÁÇÊ‚(:¼ ™àëÒñûÌ–( proc.memory.availableMB‚ 742.66015625 disk.sda.in‚0 disk.sda3.out‚ 132.515625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 8823.4453125% disk.nvme0n1.out‚14359.7314453125 disk.sda1.out‚0' disk.nvme0c0n1.out‚14359.7314453125 proc.cpu.threads‚17 network.sent‚ 7.847239e+06 disk.rbd0.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2720.7265625 disk.rbd1.in‚0 disk.rbd1.out‚ 10.64453125 disk.rbd0.in‚0 disk.dm-1.out‚ 132.515625" disk.nvme0c0n1.in‚ 8823.4453125" disk.dm-0.out‚14359.7314453125 disk.sda.out‚ 132.515625 disk.sda3.in‚0 network.recv‚ 1.166994e+06$ disk./.usageGB‚596.6966285705566 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚95.46716213226318) proc.memory.percent‚16.60599708557129( disk./.usagePercent‚40.0448114051889 disk.drbd2165.in‚0 cpu‚26.049165927044637 disk.sda1.in‚0 disk.dm-0.in‚ 8823.4453125y‘Ë‚(:† ¨àëÒöî“ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç EÅÌ‚(:º ¨àëÒ Û¥–! disk.nvme0n1.in‚ 8964.70703125% disk.nvme0n1.out‚14366.8837890625 disk.rbd1.out‚ 10.70703125 disk.dm-1.out‚ 132.78125# disk.nvme0c0n1.in‚ 8964.70703125) proc.memory.percent‚16.60599708557129 cpu‚26.050566911315787 disk.sda.in‚0 network.recv‚ 1.172171e+06) disk./.usagePercent‚40.04481908541493' disk.nvme0c0n1.out‚14366.8837890625 disk.sda3.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda2.out‚0$ memory_percent‚95.46713829040527 disk.sda.out‚ 132.78125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 7.9102e+06 disk.sda2.in‚0 disk.sda3.out‚ 132.78125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚596.6967430114746 disk.dm-0.in‚ 8964.70703125" disk.dm-0.out‚14366.8837890625 proc.cpu.threads‚17' proc.memory.availableMB‚ 742.6640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 Ë<¢“Í‚(:ˆ ·àëÒ¦‰• gpu.0.gpu‚97.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿eÅ΂(:º ·àëÒΤú•# memory_percent‚95.4674243927002 disk.rbd1.out‚ 10.70703125 disk.nvme0n1.in‚ 8965.671875$ disk./.usageGB‚596.6967964172363 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.60599708557129) disk./.usagePercent‚40.04482266952041 disk.rbd0.out‚0 disk.sda.out‚ 133.3359375! disk.nvme0c0n1.in‚ 8965.671875 cpu‚26.05161118323763 proc.cpu.threads‚17' proc.memory.availableMB‚ 742.6171875 network.recv‚ 1.175762e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 133.3359375 disk.sda2.in‚0% disk.nvme0n1.out‚14376.4638671875 disk.dm-1.in‚0" disk.dm-0.out‚14376.4638671875 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 133.3359375" proc.memory.rssMB‚ 2720.7265625 network.sent‚ 7.915244e+06 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚14376.4638671875 disk.dm-0.in‚ 8965.671875w™`2Ï- _wandbruntime‚4294  _runtime‚4294’‚·:ÈР' lr-AdamW/pg2‚0.00038956862993752225 trainer/global_step‚5049 $ _timestamp‚1.784344635429818e+09 _runtime‚4294.123558108 _step‚412 ' lr-AdamW/pg1‚0.00038956862993752225?h^eÍÑÇ _step‚412 ' lr-AdamW/pg1‚0.00038956862993752225 ' lr-AdamW/pg2‚0.00038956862993752225 trainer/global_step‚5049 $ _timestamp‚1.784344635429818e+09 _runtime‚4294.123558108œˆ±ì†2Ò- _wandbruntime‚4294  _runtime‚42947«éâ'ÓÜO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10855467 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174520585969152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325833070978 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5333635147776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5390449939968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39993452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750318 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174520695255552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19704103 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458537472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409986 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743772 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4294.124811654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843446354310977e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740341 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772091816915968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458537472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5333635147776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762818118299648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19703835 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740341 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169130154641408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5049 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733793 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766757823389696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5390540614144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39993454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5333976486912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816915968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750318 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10855732 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5049 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733793 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766757823389696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5390540614144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39993454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5333976486912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816915968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750318 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10855732 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458537472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409986 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743772 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4294.124811654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843446354310977e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740341sp†œ2Õ- _wandbruntime‚4294  _runtime‚4294nÄ¢¨úÖåM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚4294.738585202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750477 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174555045556224 PDeviceSta›P[ñtsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760470 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19707645 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5334989121024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647774720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19707903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚772240070579712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574810 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766905081458688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169163583399936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5391565144064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762964628971520ȯð&×êM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5391565144064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762964628971520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚4294.738585202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750477 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174555045556224 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760470 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19707645 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5334989121024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647774720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19707903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚772240070579712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574810 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766905081458688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169163583399936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983žñ„Êâ2Ø- _wandbruntime‚4294  _runtime‚42942ŸwîÙ 2loss/SRqqt1/assignment_loss‚0.5823488750542816 1loss/FBt1/assignment_loss‚0.02161456089153262 /loss/FBt2/detection_loss‚0.3674345087700153 %loss/total_loss‚8.629646515167297 1loss/SRqqt2/detection_loss‚0.6578633935629007 /loss/FBt1/detection_loss‚0.8653950123737256 trainer/global_step‚5049 _step‚415 epoch‚12 0loss/FRt1/assignment_loss‚1.7895454331226288 0loss/FRt2/assignment_loss‚0.9494554848087077 /loss/FRt1/detection_loss‚0.5826668637643228 .loss/FRt2/detection_loss‚1.260786008684687 % _timestamp‚1.7843446360478487e+09 _runtime‚4294.741171723 2loss/SRqqt2/assignment_loss‚0.4923384348681228 1loss/FBt2/assignment_loss‚0.04387785688084037 1loss/SRqqt1/detection_loss‚1.0163200823855305ÝàÕY Ú† 1loss/SRqqt2/detection_loss‚0.6578633935629007 /loss/FBt1/detection_loss‚0.8653950123737256 trainer/global_step‚5049 _step‚415 epoch‚12 0loss/FRt1/assignment_loss‚1.7895454331226288 0loss/FRt2/assignment_loss‚0.9494554848087077 /loss/FRt1/detection_loss‚0.5826668637643228 .loss/FRt2/detection_loss‚1.260786008684687 % _timestamp‚1.7843446360478487e+09 _runtime‚4294.741171723 2loss/SRqqt2/assignment_loss‚0.4923384348681228 1loss/FBt2/assignment_loss‚0.04387785688084037 1loss/SRqqt1/detection_loss‚1.0163200823855305 2loss/SRqqt1/assignment_loss‚0.5823488750542816 1loss/FBt1/assignment_loss‚0.02161456089153262 /loss/FBt2/detection_loss‚0.3674345087700153 %loss/total_loss‚8.629646515167297Ÿ.t·Û‚(:’ ÆàëÒÞý®• gpu.0.gpu‚77.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}8¸Ü‚(:­ ÆàëÒÈù — network.sent‚ 7.920068e+06 disk.sda3.in‚0 disk.sda3.out‚ 133.703125 network.recv‚ 1.17915e+06 disk.sda1.in‚0 disk.dm-0.in‚ 8965.671875 disk.nvme0n1.in‚ 8965.671875& proc.memory.availableMB‚ 742.765625" proc.memory.rssMB‚ 2720.7265625 cpu‚26.052685293699493# disk./.usageGB‚596.696891784668 disk.rbd1.out‚ 10.76953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 133.703125 disk.drbd2165.in‚0 disk.dm-0.out‚ 14383.2421875 disk.sda.out‚ 133.703125! disk.nvme0c0n1.in‚ 8965.671875 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0n1.out‚ 14383.2421875 disk.rbd1.in‚0) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 14383.2421875) disk./.usagePercent‚40.04482906970877 disk.sda2.out‚0$ memory_percent‚95.46651840209961 disk.rbd0.out‚0Þ~× ‘Ý‚(:† ÕàëÒíó¬” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v©#sÕÞ‚(:Ê ÕàëÒ©­ê• disk.sda.out‚ 135.69140625 disk.sda3.in‚0( proc.memory.availableMB‚ 742.58203125 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8966.50390625 disk.dm-0.in‚ 8966.50390625& disk.nvme0n1.out‚14401.11279296875$ memory_percent‚95.46763896942139 disk.drbd2165.out‚0 network.sent‚ 7.982987e+06 network.recv‚ 1.184827e+06$ disk./.usageGB‚596.7016258239746 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 cpu‚26.053613673551084 proc.cpu.threads‚17 disk.rbd1.out‚ 10.76953125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14401.11279296875# disk.dm-0.out‚14401.11279296875 disk.sda2.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda1.out‚0 disk.dm-1.out‚ 135.69140625* disk./.usagePercent‚40.045146775058754! disk.nvme0n1.in‚ 8966.50390625 disk.sda3.out‚ 135.69140625 disk.rbd1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.60599708557129|«“Õ2ß- _wandbruntime‚4334  _runtime‚4334F}üÆàÀ & lr-AdamW/pg1‚0.0003883431050207165 & lr-AdamW/pg2‚0.0003883431050207165 trainer/global_step‚5099 $ _timestamp‚1.784344675998767e+09 _runtime‚4334.692350451 _step‚416 a¡ËáÅ & lr-AdamW/pg1‚0.0003883431050207165 & lr-AdamW/pg2‚0.0003883431050207165 trainer/global_step‚5099 $ _timestamp‚1.784344675998767e+09 _runtime‚4334.692350451 _step‚416 8=nÙ2â- _wandbruntime‚4334  _runtime‚4334ç}½È&ãÛO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5441742289920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5384330649600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19894912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19895192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚779504563899392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774119891910656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176243225571328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770143651893248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774119874871296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47194793 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚5099 $ _timestamp‚1.784344675999895e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671988736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚779504563899392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10959294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490494 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5376109007996 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330649600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40375052 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205520896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774119874871296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176243334857728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204770 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205520896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774119874871296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176243334857728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204770 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10959294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490494 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5376109007996 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330649600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40375052 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19898718 $ _timestamp‚1.784344676614132e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176277782936064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770290179604480 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8937363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817563136 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770290162565120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596522 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19898987 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779652817563136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176277665268224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459177472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40382684 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10961621 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170835021326336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47213836 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚774267115900928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684622848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203845 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5385343276544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5442761609728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87586529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚418 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5099 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10961355 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40382686 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8937366 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 _runtime‚4335.307852404 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304)) ½ð&éêM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40382686 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8937366 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 _runtime‚4335.307852404 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19898718 $ _timestamp‚1.784344676614132e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176277782936064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770290179604480 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8937363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817563136 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770290162565120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596522 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19898987 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779652817563136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176277665268224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459177472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40382684 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10961621 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170835021326336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47213836 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚774267115900928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684622848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203845 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5385343276544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5442761609728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87586529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚418 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5099 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10961355 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288¢ö[éj2ê- _wandbruntime‚4335  _runtime‚4335ém|É ë„ 3loss/FBt1/assignment_loss‚0.0005601426727559881 /loss/FBt1/detection_loss‚0.7378077449869703 %loss/total_loss‚8.482528940890894 trainer/global_step‚5099 _runtime‚4335.309934084 _step‚419 2loss/SRqqt1/assignment_loss‚0.5647971227259974 /loss/FRt2/detection_loss‚1.3449121187903397 /loss/FBt2/detection_loss‚0.6026958011377316 % _timestamp‚1.7843446766165605e+09 0loss/FRt1/assignment_loss‚1.7022040397173068 2loss/FBt2/assignment_loss‚0.010791704392242871 /loss/FRt1/detection_loss‚0.5725396026461201 0loss/SRqqt2/detection_loss‚0.652277312878595 2loss/SRqqt2/assignment_loss‚0.5187011770013703 1loss/SRqqt1/detection_loss‚0.9170526313561812 epoch‚12 0loss/FRt2/assignment_loss‚0.8581895425852843Ùíì‰IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367490560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20085591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571002 $ _timestamp‚1.784344716528473e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20085315 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚5149 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172473011881984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5492822487552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405897 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40756652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177965852981760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚4375.222095993 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367490560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952504320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659222 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177965852981760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚4375.222095993 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367490560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952504320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659222 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367490560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20085591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571002 $ _timestamp‚1.784344716528473e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20085315 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚5149 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172473011881984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5492822487552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208¥¶²ú2÷- _wandbruntime‚4375  _runtime‚4375¾!ÈÌëøæM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11064439 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5493750656512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178000300576256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20089052 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5493841323520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚422 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11064708 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777615696158720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚787065206160896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47668288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 _runtime‚4375.834757777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9024616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5436038778368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_bÓ_–'atch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843447171410217e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5436038778368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658297 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783043790296708 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707346–[Úñ&ùëM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9024616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5436038778368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843447171410217e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5436038778368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658297 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783043790296708 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707346 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11064439 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5493750656512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178000300576256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20089052 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5493841323520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚422 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11064708 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777615696158720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚787065206160896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47668288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 _runtime‚4375.834757777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¦4FÄ2ú- _wandbruntime‚4375  _runtime‚4375:ñÚû 0loss/FRt1/assignment_loss‚1.7317277078945486 0loss/FBt2/assignment_loss‚0.0703909993812198 /loss/FRt1/detection_loss‚0.5729950016696075 1loss/SRqqt1/detection_loss‚0.9925411449621344 /loss/FBt2/detection_loss‚0.5781244058099437 epoch‚12 % _timestamp‚1.7843447171448443e+09 2loss/SRqqt1/assignment_loss‚0.5996051476155352 2loss/SRqqt2/assignment_loss‚0.4979841769304499 /loss/FRt2/detection_loss‚1.3897897434300222 1loss/SRqqt2/detection_loss‚0.6009866306949457 .loss/FBt1/detection_loss‚0.655407574372711 trainer/global_step‚5149 _runtime‚4375.838122808 _step‚423 0loss/FRt2/assignment_loss‚0.8587358414870987 2loss/FBt1/assignment_loss‚0.000917709292186449 %loss/total_loss‚8.549206083540403åŒ( ü† 0loss/FBt2/assignment_loss‚0.0703909993812198 /loss/FRt1/detection_loss‚0.5729950016696075 1loss/SRqqt1/detection_loss‚0.9925411449621344 /loss/FBt2/detection_loss‚0.5781244058099437 epoch‚12 % _timestamp‚1.7843447171448443e+09 2loss/SRqqt1/assignment_loss‚0.5996051476155352 2loss/SRqqt2/assignment_loss‚0.4979841769304499 /loss/FRt2/detection_loss‚1.3897897434300222 1loss/SRqqt2/detection_loss‚0.6009866306949457 .loss/FBt1/detection_loss‚0.655407574372711 trainer/global_step‚5149 _runtime‚4375.838122808 _step‚423 0loss/FRt2/assignment_loss‚0.8587358414870987 2loss/FBt1/assignment_loss‚0.000917709292186449 %loss/total_loss‚8.549206083540403 0loss/FRt1/assignment_loss‚1.7317277078945486§ü`³_ý‚(:’ ‘áëÒ¶ìú• gpu.0.gpu‚88.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚283.228' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.64560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò&ºbÑþ‚(:Æ ‘áëÒÅ¢™— network.sent‚ 8.11043e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 9105.17578125( disk.nvme0c0n1.out‚14594.21826171875 disk.dm-0.in‚ 9105.17578125& disk.nvme0n1.out‚14594.21826171875 disk.dm-1.out‚ 137.32421875& proc.memory.availableMB‚ 496.453125" proc.memory.rssMB‚ 2720.7265625 disk.rbd0.in‚0# disk.dm-0.out‚14594.21826171875) proc.memory.percent‚16.60599708557129 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 137.32421875 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚96.96989059448242$ disk./.usageGB‚596.7853317260742 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.95703125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 9105.17578125 disk.sda3.in‚0 network.recv‚ 1.20121e+06* disk./.usagePercent‚40.050764348382586 disk.sda.out‚ 137.32421875 cpu‚26.057913976552168 du“ÿ‚(:ˆ  áëҌӤ“ gpu.0.gpu‚65.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µéÑ„Í€‚(:  áëÒøÛè— disk.nvme0n1.in‚ 9230.703125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 496.1640625" proc.memory.rssMB‚ 2720.7265625 proc.cpu.threads‚17 disk.sda1.out‚0# disk.dm-0.out‚14670.82275390625 network.sent‚ 8.162434e+06$ disk./.usageGB‚596.7849349975586 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 9230.703125$ memory_percent‚96.97165489196777) proc.memory.percent‚16.60599708557129 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 137.47265625! disk.nvme0c0n1.in‚ 9230.703125 cpu‚26.059179431981715 network.recv‚ 1.206045e+06 disk.sda3.in‚0 disk.rbd1.out‚ 11.01953125) disk./.usagePercent‚40.05073772359903& disk.nvme0n1.out‚14670.82275390625 disk.rbd1.in‚0 disk.sda.out‚ 137.47265625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 137.47265625( disk.nvme0c0n1.out‚14670.82275390625+?—jž‚(:“ ¯áëÒ¥š½“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚147.075' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.414999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðjWHÓ‚‚(:È ¯áëÒÙ…ö• disk.rbd0.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚14680.81884765625 disk.sda1.in‚0' proc.memory.availableMB‚ 496.1484375 disk.dm-1.in‚0 disk.dm-1.out‚ 137.72265625 disk.drbd2165.out‚0) proc.memory.percent‚16.60599708557129( disk.nvme0c0n1.out‚14680.81884765625 disk.sda.out‚ 137.72265625$ memory_percent‚96.97175025939941 disk.rbd1.in‚0 network.sent‚ 8.165398e+06 disk.sda3.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚596.7849578857422 disk.dm-0.in‚ 9231.53515625 disk.rbd1.out‚ 11.01953125) disk./.usagePercent‚40.05073925964424 disk.sda2.in‚0! disk.nvme0n1.in‚ 9231.53515625# disk.dm-0.out‚14680.81884765625 disk.sda.in‚0 cpu‚26.060072742425277 network.recv‚ 1.209604e+06 disk.rbd0.out‚0 disk.sda3.out‚ 137.72265625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 9231.53515625" proc.memory.rssMB‚ 2720.7265625‘{á*2ƒ- _wandbruntime‚4415  _runtime‚4415 5|ÎÇ„Á trainer/global_step‚5199 % _timestamp‚1.7843447571787443e+09 _runtime‚4415.872316727 _step‚424 & lr-AdamW/pg1‚0.0003858605719794316 & lr-AdamW/pg2‚0.0003858605719794316Ù_Ì…Æ trainer/global_step‚5199 % _timestamp‚1.7843447571787443e+09 _runtime‚4415.872316727 _step‚424 & lr-AdamW/pg1‚0.0003858605719794316 & lr-AdamW/pg2‚0.0003858605719794316¨ âJõ2†- _wandbruntime‚4415  _runtime‚4415 GÀúá'‡ÛO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9110118 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89241949 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚788843994873856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48113674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179688201566720 trainer/global_step‚5199 $ _timestamp‚1.784344757179805e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11164628 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5543838238208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5477341981812 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5485721653248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11164902 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103697 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721653248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41138254 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788843994873856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚425 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794329699487744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41138252 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48103697 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚4415.873544965 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5486062992384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41138254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784794719080448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5477682531702 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5543751758336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41138252 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788843977834496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179688306658816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20274746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20275023 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057866240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174144468420608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5486062992384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174144449808384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚790272061062260 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699487744 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179688306658816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20274746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20275023 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057866240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174144468420608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5486062992384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174144449808384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚790272061062260 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699487744 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576©˜Rgx2‰- _wandbruntime‚4416  _runtime‚44161ݶì&ŠæM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48112749 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9111863 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41145886 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268626 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5544862078976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5478353035534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843447577937365e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258633 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5486734280192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311529984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122740 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11166675 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179722647881728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174177878566912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5544769314816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784941229752320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174177897179136 _step‚426 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075626496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788991235903488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268626 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179722759258112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11166949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953144320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41145884 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311529984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48112749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚4416.487483102 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585536 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784941229752320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174177897179136 _step‚426 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075626496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788991235903488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268626 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179722759258112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11166949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953144320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41145884 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311529984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48112749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚4416.487483102 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585536 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268626 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5544862078976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5478353035534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843447577937365e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258633 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5486734280192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311529984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122740 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11166675 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179722647881728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174177878566912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ªȨ̈2Œ- _wandbruntime‚4416  _runtime‚44164|â ƒ /loss/FRt1/assignment_loss‚1.856057678853666 0loss/FRt2/assignment_loss‚0.7787701127879909 3loss/FBt1/assignment_loss‚0.0006271669518287126 1loss/FBt2/assignment_loss‚0.01604712744717143 0loss/FBt2/detection_loss‚0.45469109249719675 epoch‚12 _runtime‚4416.490500358 2loss/SRqqt2/assignment_loss‚0.5179630027213085 %loss/total_loss‚8.704564223518291 % _timestamp‚1.7843447577971854e+09 /loss/FRt1/detection_loss‚0.5619856005410562 /loss/FRt2/detection_loss‚1.4490056389395503 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.5429671465923699 0loss/SRqqt1/detection_loss‚0.883388364078395 1loss/SRqqt2/detection_loss‚0.6825176269296375 /loss/FBt1/detection_loss‚0.9605436651781202 _step‚427Ö– ¥Žˆ 3loss/FBt1/assignment_loss‚0.0006271669518287126 1loss/FBt2/assignment_loss‚0.01604712744717143 0loss/FBt2/detection_loss‚0.45469109249719675 epoch‚12 _runtime‚4416.490500358 2loss/SRqqt2/assignment_loss‚0.5179630027213085 %loss/total_loss‚8.704564223518291 % _timestamp‚1.7843447577971854e+09 /loss/FRt1/detection_loss‚0.5619856005410562 /loss/FRt2/detection_loss‚1.4490056389395503 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.5429671465923699 0loss/SRqqt1/detection_loss‚0.883388364078395 1loss/SRqqt2/detection_loss‚0.6825176269296375 /loss/FBt1/detection_loss‚0.9605436651781202 _step‚427 /loss/FRt1/assignment_loss‚1.856057678853666 0loss/FRt2/assignment_loss‚0.7787701127879909«Uï4’‚(:‡ ¾áëÒõ餓 gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ö¼‚(:± ¾áëÒÿ©Ñ– disk.sda2.out‚0# disk.dm-0.out‚14762.70263671875 disk.dm-1.in‚0 cpu‚26.067664230119625 network.sent‚ 8.17038e+06) disk./.usagePercent‚40.05072748329766# disk./.usageGB‚596.784782409668 disk.rbd0.in‚0 disk.nvme0c0n1.in‚9232.5 disk.sda.in‚0 disk.sda.out‚ 138.05859375 disk.sda2.in‚0& disk.nvme0n1.out‚14762.70263671875 disk.sda3.in‚0 disk.dm-1.out‚ 138.05859375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚14762.70263671875 disk.sda3.out‚ 138.05859375 disk.rbd0.out‚0 disk.dm-0.in‚9232.5' proc.memory.availableMB‚ 496.1328125 proc.cpu.threads‚17 disk.sda1.in‚0$ memory_percent‚96.97184562683105 network.recv‚ 1.212957e+06 disk.sda1.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2720.7265625) proc.memory.percent‚16.60599708557129 disk.nvme0n1.in‚9232.5 disk.rbd1.in‚0 disk.rbd1.out‚ 11.08203125¤— ³2‘- _wandbruntime‚4434  _runtime‚4434+÷É80’À{ >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08630678802728653 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.875990629196167 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27806520462036133 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33553966879844666 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 'particle/f_score‚0.6163052916526794 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 jet/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19876816868782043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32067060470581055 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7126103043556213 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7069878578186035 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981451630592346 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5702766180038452 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22769467532634735 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11098126322031021 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.12904267013072968 *jet/accuracy_1_of_3‚0.9184749126434326 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29670000076293945 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992965042591095 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917654991149902 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8053821325302124 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9811909198760986 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7385552525520325 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0242299921810627 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24259303510189056 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6905204057693481 7 Purity/*FRt1SRqqt2FBt/FBt_puriA1JØ–.ty‚0.9351851344108582 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8193685412406921 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02471354231238365 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_3_of_3‚0.04309117794036865 particle/accuracy_1_of_4‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28505781292915344 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2199515998363495 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0394279919564724 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.653595507144928 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7572388648986816 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9147776365280151 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4808169901371002 particle/accuracy_1_of_5‚1 )loss/val_total_loss‚8.232602292895884 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944285154342651 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06662647426128387 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02356947399675846 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11750000715255737 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23092953860759735 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5145829916000366 *jet/accuracy_2_of_4‚0.7541077136993408 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858404994010925 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5763700604438782 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02427440509200096 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7723166942596436 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04500216618180275 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6814732551574707 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_5‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6727222800254822 _runtime‚4434.498065775 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9716809391975403 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.852985680103302 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46669116616249084 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7497961521148682 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0392875038087368 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2399049550294876 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8046828508377075 epoch‚12 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6422228217124939 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09393629431724548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6579707264900208 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7450491786003113 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8681904673576355 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944211840629578 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_2‚0.7784526348114014 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6579707264900208 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8283258080482483 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2185772955417633 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 &particle/accuracy‚0.83842933177948 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830886125564575 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9889722466468811 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2915010154247284 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2591487467288971 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416269302368 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9369623064994812 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609600067138672 +particle/specificity‚0.9024766087532043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6926572918891907 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6407216191291809 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21853330731391907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6598910689353943 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07530353963375092 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19557428359985352 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27817100286483765 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16983965039253235 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6651617884635925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932611584663391 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28723427653312683 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 *jet/accuracy_1_of_1‚0.3533901274204254 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6343716979026794 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7501026391983032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28723427653312683 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9403979778289795 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973406195640564 +particle/sensitivity‚0.6042525172233582 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1086064949631691 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6632915735244751 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03811868652701378 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7052269577980042 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4740084707736969 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8470332622528076 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7381685972213745 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025868432596325874 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7505196928977966 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.827626645565033 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6699649095535278 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8681904673576355 ,jet/accuracy_3_of_4‚0.042860183864831924 _step‚428 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23888665437698364 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21503843367099762 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2158675193786621 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2982746362686157 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.38380104303359985 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3456694185733795 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6065618991851807 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877877235412598 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2903708517551422 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37743183970451355 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7787535786628723 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7491622567176819 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7503602504730225 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 % _timestamp‚1.7843447758031514e+09 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1261797398328781 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.784565269947052 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954596757888794 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6577410101890564 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025868432596325874 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21657457947731018 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 trainer/global_step‚5199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6925679445266724 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7781810760498047 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7901304960250854 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49492013454437256 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6987602710723877 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3889091908931732 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20152083039283752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 )jet/accuracy_1_of_4‚0.983000636100769‡ð ÖË=“Å{ 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49492013454437256 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6987602710723877 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3889091908931732 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20152083039283752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 )jet/accuracy_1_of_4‚0.983000636100769 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08630678802728653 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.875990629196167 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27806520462036133 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33553966879844666 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 'particle/f_score‚0.6163052916526794 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 jet/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19876816868782043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32067060470581055 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7126103043556213 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7069878578186035 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981451630592346 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5702766180038452 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22769467532634735 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11098126322031021 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.12904267013072968 *jet/accuracy_1_of_3‚0.9184749126434326 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29670000076293945 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992965042591095 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917654991149902 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8053821325302124 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9811909198760986 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7385552525520325 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0242299921810627 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24259303510189056 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6905204057693481 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8193685412406921 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02471354231238365 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_3_of_3‚0.04309117794036865 particle/accuracy_1_of_4‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28505781292915344 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2199515998363495 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0394279919564724 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.653595507144928 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7572388648986816 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9147776365280151 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4808169901371002 particle/accuracy_1_of_5‚1 )loss/val_total_loss‚8.232602292895884 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944285154342651 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06662647426128387 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02356947399675846 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11750000715255737 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23092953860759735 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5145829916000366 *jet/accuracy_2_of_4‚0.7541077136993408 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858404994010925 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5763700604438782 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02427440509200096 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7723166942596436 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04500216618180275 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6814732551574707 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_5‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6727222800254822 _runtime‚4434.498065775 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9716809391975403 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.852985680103302 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46669116616249084 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7497961521148682 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0392875038087368 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2399049550294876 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8046828508377075 epoch‚12 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6422228217124939 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09393629431724548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6579707264900208 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7450491786003113 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8681904673576355 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944211840629578 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_2‚0.7784526348114014 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6579707264900208 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8283258080482483 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2185772955417633 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 &particle/accuracy‚0.83842933177948 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830886125564575 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9889722466468811 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2915010154247284 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2591487467288971 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416269302368 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9369623064994812 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609600067138672 +particle/specificity‚0.9024766087532043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6926572918891907 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6407216191291809 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21853330731391907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6598910689353943 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07530353963375092 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19557428359985352 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27817100286483765 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16983965039253235 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6651617884635925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932611584663391 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28723427653312683 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 *jet/accuracy_1_of_1‚0.3533901274204254 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6343716979026794 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7501026391983032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28723427653312683 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9403979778289795 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973406195640564 +particle/sensitivity‚0.6042525172233582 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1086064949631691 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6632915735244751 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03811868652701378 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7052269577980042 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4740084707736969 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8470332622528076 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7381685972213745 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025868432596325874 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7505196928977966 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.827626645565033 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6699649095535278 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8681904673576355 ,jet/accuracy_3_of_4‚0.042860183864831924 _step‚428 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23888665437698364 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21503843367099762 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2158675193786621 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2982746362686157 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.38380104303359985 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3456694185733795 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6065618991851807 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877877235412598 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2903708517551422 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37743183970451355 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7787535786628723 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7491622567176819 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7503602504730225 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 % _timestamp‚1.7843447758031514e+09 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1261797398328781 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.784565269947052 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954596757888794 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6577410101890564 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025868432596325874 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21657457947731018 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 trainer/global_step‚5199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6925679445266724 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7781810760498047 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7901304960250854¬º"€@“”‚(:ˆ ÍáëÒöÆ” gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z8ƒ¶¹•‚(:® ÍáëÒþ÷»– disk.nvme0c0n1.in‚9232.5 disk.sda.in‚0 disk.rbd1.out‚ 11.08203125$ memory_percent‚95.46144008636475 disk.sda2.in‚0) disk./.usagePercent‚40.05078534100039 disk.dm-1.out‚ 138.48828125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚15011.2236328125" disk.dm-0.out‚15011.2236328125 disk.sda2.out‚0 disk.nvme0n1.in‚9232.5 network.recv‚ 1.218429e+06" disk./.usageGB‚596.78564453125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 138.48828125 disk.sda3.in‚0( proc.memory.availableMB‚ 743.59765625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚9232.5) proc.memory.percent‚16.60599708557129 cpu‚26.074655832167448 proc.cpu.threads‚17 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.sda3.out‚ 138.48828125 disk.dm-1.in‚0% disk.nvme0n1.out‚15011.2236328125 network.sent‚ 8.23322e+06 disk.rbd0.in‚0Ï›:Å“–‚(:ˆ ÜáëÒ’öð” gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï_öׂ(:¸ ÜáëÒïÙÁ– disk.dm-0.in‚ 9338.87109375 disk.rbd1.out‚ 11.14453125 disk.dm-1.in‚0( proc.memory.availableMB‚ 743.33984375" proc.memory.rssMB‚ 2720.7265625 disk.sda3.out‚138.875 disk.sda1.out‚0 cpu‚26.07552058156552 network.recv‚ 1.223264e+06) proc.memory.percent‚16.60599708557129 network.sent‚ 8.29068e+06 disk.rbd0.out‚0 disk.dm-1.out‚138.875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚15090.27587890625# disk.dm-0.out‚15090.27587890625! disk.nvme0n1.in‚ 9338.87109375$ memory_percent‚95.46301364898682$ disk./.usageGB‚596.7856330871582 disk.sda.out‚138.875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚15090.27587890625 disk.sda3.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚40.05078457297779# disk.nvme0c0n1.in‚ 9338.87109375 disk.sda2.out‚0 proc.cpu.threads‚17qÏ,~ž˜‚(:“ ëáëÒÕ›ù” gpu.0.gpu‚81.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚183.407' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.681400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ“@¼Î™‚(:à ëáëÒèÒ§– network.recv‚ 1.226826e+06 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 139.11328125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.14453125& proc.memory.availableMB‚ 743.515625) proc.memory.percent‚16.60599708557129 disk.sda1.in‚0 disk.sda3.out‚ 139.11328125 cpu‚26.07660519983683 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 9338.87109375! disk.nvme0n1.in‚ 9338.87109375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9338.87109375 network.sent‚ 8.2956e+06# disk./.usageGB‚596.784610748291 disk.dm-1.out‚ 139.11328125 disk.drbd2165.in‚0& disk.nvme0n1.out‚15338.07080078125) disk./.usagePercent‚40.05071596295862 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚15338.07080078125 disk.sda2.out‚0" proc.memory.rssMB‚ 2720.7265625 disk.rbd0.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚15338.07080078125$ memory_percent‚95.46194076538086Ñ[& 2š- _wandbruntime‚4476  _runtime‚4476Zòcɛà ' lr-AdamW/pg1‚0.00038460372330341067 ' lr-AdamW/pg2‚0.00038460372330341067 trainer/global_step‚5249 % _timestamp‚1.7843448177736938e+09 _runtime‚4476.467337534 _step‚429Ÿj‚>ÎœÈ ' lr-AdamW/pg1‚0.00038460372330341067 ' lr-AdamW/pg2‚0.00038460372330341067 trainer/global_step‚5249 % _timestamp‚1.7843448177736938e+09 _runtime‚4476.467337534 _step‚429­*šL2- _wandbruntime‚4476  _runtime‚4476Vt…;žÚO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628476 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.jìb‡¥&small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5554150363136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488 _runtime‚ 4476.46850093 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181445118392320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175831799834624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48628476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175831818446848 % _timestamp‚1.7843448177747715e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41640520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48638453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11299562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5613219757056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803690104708096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20499190 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181445019591680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚794028915888128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491702272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798135595966464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5613299945472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743844 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11299302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193954 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20499453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199891 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671332 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90278973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491702272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154T ùå'ŸßO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11299302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193954 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20499453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199891 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671332 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90278973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491702272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628476 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5554150363136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488 _runtime‚ 4476.46850093 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181445118392320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175831799834624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48628476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175831818446848 % _timestamp‚1.7843448177747715e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41640520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48638453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11299562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5613219757056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803690104708096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20499190 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181445019591680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚794028915888128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491702272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798135595966464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5613299945472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743844 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872®¾Ñ2 - _wandbruntime‚4477  _runtime‚4477ÌPì&¡æM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175865228593152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41648150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546689247676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555162990080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5614243642880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚431 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48637528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843448183894174e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803838358371840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647519 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11301331 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20503228 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181479575223808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999986176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803838358371840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 trainer/global_step‚5249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162990080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20502964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664NÓ‰"5 ¢ëM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20503228 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181479575223808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999986176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803838358371840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_tr㽺ȼain_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 trainer/global_step‚5249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162990080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20502964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175865228593152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41648150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546689247676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555162990080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5614243642880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚431 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48637528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843448183894174e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803838358371840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647519 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11301331 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0¯ÊpHD2£- _wandbruntime‚4477  _runtime‚4477)°~⤠%loss/total_loss‚8.649784016152278 2loss/SRqqt1/assignment_loss‚0.5485975158004306 0loss/SRqqt2/assignment_loss‚0.51989062131154 3loss/FBt1/assignment_loss‚0.0025117334194654827 trainer/global_step‚5249 _runtime‚4477.085235089 _step‚432 epoch‚13 .loss/FRt1/detection_loss‚0.586743210734593 /loss/FRt2/detection_loss‚1.2577891392470608 /loss/FBt2/detection_loss‚0.5475646675654476 % _timestamp‚1.7843448183919477e+09 0loss/FRt2/assignment_loss‚0.9257379478009051 /loss/FBt1/detection_loss‚0.9163431290250558 /loss/FRt1/assignment_loss‚1.694499777062401 2loss/FBt2/assignment_loss‚0.007947981158573217 1loss/SRqqt1/detection_loss‚1.0270956136298635 1loss/SRqqt2/detection_loss‚0.6150626793969423-äþ¼ ¥† /loss/FBt2/detection_loss‚0.5475646675654476 % _timestamp‚1.7843448183919477e+09 0loss/FRt2/assignment_loss‚0.9257379478009051 /loss/FBt1/detection_loss‚0.9163431290250558 /loss/FRt1/assignment_loss‚1.694499777062401 2loss/FBt2/assignment_loss‚0.007947981158573217 1loss/SRqqt1/detection_loss‚1.0270956136298635 1loss/SRqqt2/detection_loss‚0.6150626793969423 %loss/total_loss‚8.649784016152278 2loss/SRqqt1/assignment_loss‚0.5485975158004306 0loss/SRqqt2/assignment_loss‚0.51989062131154 3loss/FBt1/assignment_loss‚0.0025117334194654827 trainer/global_step‚5249 _runtime‚4477.085235089 _step‚432 epoch‚13 .loss/FRt1/detection_loss‚0.586743210734593 /loss/FRt2/detection_loss‚1.2577891392470608°«ÊÔ_’¦‚(:‡ úáëÒÝŠ ” gpu.0.gpu‚63.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô“ªÑ§‚(:Æ úáëÒþÏ›–* disk./.usagePercent‚40.050718779041496 disk.rbd1.in‚0 disk.rbd0.out‚0# memory_percent‚95.4639196395874 network.recv‚ 1.230242e+06 disk.sda3.out‚ 139.55859375 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 9339.8359375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚15417.60986328125 cpu‚26.077182349190345$ disk./.usageGB‚596.7846527099609 disk.rbd1.out‚ 11.20703125" proc.memory.rssMB‚ 2720.7265625) proc.memory.percent‚16.60599708557129 network.sent‚ 8.300567e+06( disk.nvme0c0n1.out‚15417.60986328125 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 139.55859375 disk.nvme0n1.in‚ 9339.8359375( proc.memory.availableMB‚ 743.19140625 disk.drbd2165.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 9339.8359375# disk.dm-0.out‚15417.60986328125 disk.sda.out‚ 139.55859375 disk.sda2.in‚0^ÄiOž¨‚(:“ ‰âëÒ­áó“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.633' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.726599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…OYGÁ©‚(:¶ ‰âëÒòÌ™– disk.rbd0.out‚0$ memory_percent‚95.46236991882324 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 9340.66796875 cpu‚26.078251700938647 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 9340.66796875 disk.sda1.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.rbd1.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 9340.66796875" disk.nvme0c0n1.out‚ 15425.03125 disk.dm-0.out‚ 15425.03125 network.sent‚ 8.363487e+06) disk./.usagePercent‚40.05072441120725 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 139.96484375 proc.cpu.threads‚17 disk.sda.out‚ 139.96484375 disk.nvme0n1.out‚ 15425.03125 disk.sda3.out‚ 139.96484375 disk.rbd1.out‚ 11.20703125' proc.memory.availableMB‚ 743.4453125) proc.memory.percent‚16.60599708557129 network.recv‚ 1.235234e+06$ disk./.usageGB‚596.7847366333008`Bꓪ‚(:ˆ ˜âëÒòÅ“• gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¸ú;Ϋ‚(:à ˜âëÒš½¯– disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 140.1796875# disk.nvme0c0n1.in‚ 9402.61328125# disk.dm-0.out‚15500.47509765625 disk.sda.in‚0" proc.memory.rssMB‚ 2720.7265625 network.recv‚ 1.238465e+06 disk.drbd2165.in‚0 disk.sda.out‚ 140.1796875 disk.sda2.in‚0) proc.memory.percent‚16.60599708557129 cpu‚26.07909203022099 disk.rbd0.in‚0 disk.dm-0.in‚ 9402.61328125! disk.nvme0n1.in‚ 9402.61328125 disk.sda3.in‚0 network.sent‚ 8.368645e+06& disk.nvme0n1.out‚15500.47509765625) disk./.usagePercent‚40.05072057109423 disk.dm-1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚15500.47509765625 disk.sda3.out‚ 140.1796875& proc.memory.availableMB‚ 743.453125$ disk./.usageGB‚596.7846794128418 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 11.26953125$ memory_percent‚95.46232223510742H{([2¬- _wandbruntime‚4516  _runtime‚45168™í;É­Ã ' lr-AdamW/pg1‚0.00038333659393536174 ' lr-AdamW/pg2‚0.00038333659393536174 trainer/global_step‚5299 % _timestamp‚1.7843448582447534e+09 _runtime‚4516.938339822 _step‚433–rE‚Î®È ' lr-AdamW/pg1‚0.00038333659393536174 ' lr-AdamW/pg2‚0.00038333659393536174 trainer/global_step‚5299 % _timestamp‚1.7843448582447534e+09 _runtime‚4516.938339822 _step‚433±< ±2¯- _wandbruntime‚4516  _runtime‚4516ˆ(°Žá'°ÛO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42022120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806951101751840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177503237761024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805497647448064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20690811 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49082928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9287141 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20690538 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11403400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493313024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183167922548224 _step‚434þ¸"±àO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42022120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806951101751840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177503237761024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805497647448064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20690811 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49082928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9287141 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20690538 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_startîÞ¾V/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11403400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493313024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183167922548224 _step‚434 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169²/"å2²- _wandbruntime‚4517  _runtime‚4517Ò"b/ì&³æM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807098623477546 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49101971 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5605858491904 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091980 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20694374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4517.553582198 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597646284586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5665595147264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5665687911424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚435 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11405491 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801500960153600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843448588597348e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091980 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801500977192960 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚811251105355264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20694374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4517.553582198 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597646284586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5665595147264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5665687911424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚435 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11405491 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801500960153600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843448588597348e+09³èîHÿ2µ- _wandbruntime‚4517  _runtime‚4517>íH¶‚ epoch‚13 trainer/global_step‚5299 % _timestamp‚1.7843448588627992e+09 0loss/FRt1/assignment_loss‚1.7595434608770344 1loss/SRqqt1/assignment_loss‚0.535771184179221 2loss/FBt1/assignment_loss‚0.020813344558811765 /loss/FRt1/detection_loss‚0.5767464566829532 /loss/FRt2/detection_loss‚1.2500663304900335 1loss/SRqqt2/detection_loss‚0.6900828296221012 _step‚436 0loss/FRt2/assignment_loss‚1.1022006302643605 2loss/SRqqt2/assignment_loss‚0.5202689996847745 2loss/FBt2/assignment_loss‚0.007182153354948658 0loss/SRqqt1/detection_loss‚0.924613637373052 /loss/FBt1/detection_loss‚0.7093389111105353 _runtime‚4517.556104229 /loss/FBt2/detection_loss‚0.5893069399900183 %loss/total_loss‚8.685934878187844ÇI³ ·‡ 2loss/FBt1/assignment_loss‚0.020813344558811765 /loss/FRt1/detection_loss‚0.5767464566829532 /loss/FRt2/detection_loss‚1.2500663304900335 1loss/SRqqt2/detection_loss‚0.6900828296221012 _step‚436 0loss/FRt2/assignment_loss‚1.1022006302643605 2loss/SRqqt2/assignment_loss‚0.5202689996847745 2loss/FBt2/assignment_loss‚0.007182153354948658 0loss/SRqqt1/detection_loss‚0.924613637373052 /loss/FBt1/detection_loss‚0.7093389111105353 _runtime‚4517.556104229 /loss/FBt2/detection_loss‚0.5893069399900183 %loss/total_loss‚8.685934878187844 epoch‚13 trainer/global_step‚5299 % _timestamp‚1.7843448588627992e+09 0loss/FRt1/assignment_loss‚1.7595434608770344 1loss/SRqqt1/assignment_loss‚0.535771184179221´ÇXÔ‘¸‚(:† §âëÒˆÕš– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ŸÀqǹ‚(:¼ §âëÒêá÷— disk.dm-0.in‚ 9425.46484375 disk.sda2.in‚0! disk.nvme0n1.in‚ 9425.46484375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 140.52734375" proc.memory.rssMB‚ 2720.7265625$ disk./.usageGB‚596.7846870422363 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 140.52734375 disk.rbd0.out‚0 disk.dm-1.out‚ 140.52734375 disk.dm-0.out‚15508.67578125) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 disk.rbd1.out‚ 11.26953125% disk.nvme0c0n1.out‚15508.67578125 disk.sda.in‚0# disk.nvme0n1.out‚15508.67578125' proc.memory.availableMB‚ 742.8359375 network.recv‚ 1.241896e+06 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 9425.46484375$ memory_percent‚95.46608924865723 network.sent‚ 8.38656e+06( disk./.usagePercent‚40.0507210831093 disk.sda2.out‚0 cpu‚26.08001946428145 disk.sda1.out‚0 disk.dm-1.in‚0“§é“º‚(:ˆ ¶âëÒ«Œ½“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûˆq›Ê»‚(:¿ ¶âëÒá—— network.recv‚ 1.246779e+06) disk./.usagePercent‚40.05175714560038 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 140.83984375# disk.dm-0.out‚15603.83349609375 disk.sda.in‚0# memory_percent‚95.4671859741211) proc.memory.percent‚16.60599708557129 disk.nvme0n1.in‚ 9426.296875 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 9426.296875 disk.sda2.out‚0& disk.nvme0n1.out‚15603.83349609375 disk.rbd0.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚15603.83349609375 disk.dm-0.in‚ 9426.296875 disk.rbd1.in‚0" proc.memory.rssMB‚ 2720.7265625 cpu‚26.080805178901787 disk.drbd2165.in‚0 disk.sda.out‚ 140.83984375 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 140.83984375 disk.sda2.in‚0 disk.rbd1.out‚ 11.39453125 disk.rbd0.out‚0% proc.memory.availableMB‚ 742.65625$ disk./.usageGB‚596.8001251220703 network.sent‚ 8.436499e+06Zmb§2¼- _wandbruntime‚4557  _runtime‚45570UeÃɽà ' lr-AdamW/pg1‚0.00038205926526063623 ' lr-AdamW/pg2‚0.00038205926526063623 trainer/global_step‚5349 % _timestamp‚1.7843448988627725e+09 _runtime‚4557.556380079 _step‚437±Ø!Î¾È _runtime‚4557.556380079 _step‚437 ' lr-AdamW/pg1‚0.00038205926526063623 ' lr-AdamW/pg2‚0.00038205926526063623 trainer/global_step‚5349 % _timestamp‚1.7843448988627725e+09µ.ûí2¿- _wandbruntime‚4557  _runtime‚4557é\)4ÀÜO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184890448809472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚814327251832348 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179174675687424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49547357 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5715861699072 trainer/global_step‚5349 _step‚438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184890555998720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91951077 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9374388 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818515598674944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42403718 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167770 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818515598674944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5655882705920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715773122048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243098 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11507080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843448988638742e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541366784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20881471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚4557.557618487 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/„Mës®reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818515240296448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818515240296448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42403718 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167770 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818515598674944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5655882705920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715773122048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243098 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11507080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843448988638742e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541366784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20881471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚4557.557618487 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¶íšJ2Â- _wandbruntime‚4558  _runtime‚4558vÝ‚„ì&ÃæM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20884993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11508860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5656895340032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20885268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49546432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚813006956998656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5647922221492 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11509132 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4558.170872821 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818663493953024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5716791117824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5656553993728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184925009037312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚813006939959296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843448994771125e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42411350 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808826510786560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136Z “¥*"ÄëM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5716885979136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚813006956998656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚814474773558054 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚439 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚813006939959296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49546432 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11509132 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4558.170872821 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818663493953024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5716791117824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5656553993728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184925009037312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚813006939959296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843448994771125e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42411350 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808826510786560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20884993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11508860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5656895340032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20885268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49546432 [GDeviceStatsMonitor.on_train_batch_end/active_b?BoÇytes.large_pool.allocated‚813006956998656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5647922221492 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42785318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11610935 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5706578207744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚5399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49991832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5706236868608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚825927987279872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50001809 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5697868204568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 $ _timestamp‚1.784344939231825e+09 _runtime‚4597.925565819 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42785318 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _step‚442 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚820221750411264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42785320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11610673 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5706578207744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9461641 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9461638 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50001809 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654678 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186613036703744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345658368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5766923089920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42785320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186613145990144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816005516668928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21072576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345658368 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11610673 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5706578207744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9461641 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9461638 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50001809 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654678 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186613036703744 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345658368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5766923089920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42785320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186613145990144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816005516668928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21072576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345658368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42785318º¹êy”2Ô- _wandbruntime‚4598  _runtime‚4598ºs;nÕæM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50000884 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21076143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826076599322112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843449398459175e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803827 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42792950 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5767948192256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820368991440896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5707249495552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249495552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11612760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21076421 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180879542372352 _runtime‚4598.539647262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9463383 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186647490564608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5707590841856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240936448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476`ª(+ñ&ÖëM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50000884 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21076143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826076599322112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843449398459175e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803827 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42792950 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5767948192256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820368991440896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5707249495552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249495552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11612760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21076421 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180879542372352 _runtime‚4598.539647262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9463383 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186647490564608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5707590841856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240936448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360»ÍàÞ2×- _wandbruntime‚4598  _runtime‚4598ÞÁ¯Ø 0loss/FRt2/assignment_loss‚0.9298599655450133 3loss/FBt1/assignment_loss‚0.0011063005364173313 1loss/FBt2/assignment_loss‚0.04879843835665854 /loss/FRt1/detection_loss‚0.5566479705279814 $ _timestamp‚1.784344939848973e+09 _step‚444 1loss/SRqqt1/detection_loss‚0.9206199815434715 1loss/SRqqt2/detection_loss‚0.6380610599929475 0loss/FRt1/assignment_loss‚1.7538821756157492 2loss/SRqqt1/assignment_loss‚0.5071651121219717 1loss/SRqqt2/assignment_loss‚0.550882283506948 .loss/FRt2/detection_loss‚1.283165937292856 /loss/FBt1/detection_loss‚0.5677151163870638 %loss/total_loss‚8.264297323652919 epoch‚13 /loss/FBt2/detection_loss‚0.5063929822258402 trainer/global_step‚5399 _runtime‚4598.542267017=a7ã Ù† 0loss/FRt1/assignment_loss‚1.7538821756157492 2loss/SRqqt1/assignment_loss‚0.5071651121219717 1loss/SRqqt2/assignment_loss‚0.550882283506948 .loss/FRt2/detection_loss‚1.283165937292856 /loss/FBt1/detection_loss‚0.5677151163870638 %loss/total_loss‚8.264297323652919 epoch‚13 /loss/FBt2/detection_loss‚0.5063929822258402 trainer/global_step‚5399 _runtime‚4598.542267017 0loss/FRt2/assignment_loss‚0.9298599655450133 3loss/FBt1/assignment_loss‚0.0011063005364173313 1loss/FBt2/assignment_loss‚0.04879843835665854 /loss/FRt1/detection_loss‚0.5566479705279814 $ _timestamp‚1.784344939848973e+09 _step‚444 1loss/SRqqt1/detection_loss‚0.9206199815434715 1loss/SRqqt2/detection_loss‚0.6380610599929475¼ÁztžÚ‚(:“ òâëÒ’Žà“ gpu.0.gpu‚92.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ÛxÔÛ‚(:É òâëÒùæõ– cpu‚26.084670847291815! disk.nvme0n1.in‚ 9502.21484375& disk.nvme0n1.out‚15630.58740234375 disk.sda3.out‚ 144.42578125 disk.rbd1.out‚ 11.51953125 network.recv‚ 1.262393e+06 disk.dm-0.in‚ 9502.21484375( proc.memory.availableMB‚ 743.49609375$ disk./.usageGB‚596.7188415527344 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 8.514282e+06 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.60599708557129) disk./.usagePercent‚40.04630213706078 disk.dm-1.out‚ 144.42578125( disk.nvme0c0n1.out‚15630.58740234375$ memory_percent‚95.46205997467041 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 144.42578125" proc.memory.rssMB‚ 2720.7265625 disk.sda1.in‚0 disk.sda.in‚0# disk.dm-0.out‚15630.58740234375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 9502.21484375ÍJJ“Ü‚(:ˆ ãëÒÙ‘Ñ“ gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹iËÉÝ‚(:¾ ãëÒ¨Íô• network.sent‚ 8.577463e+06 disk.sda.out‚ 145.078125 disk.rbd1.out‚ 11.51953125 disk.dm-1.out‚ 145.078125" disk.dm-0.out‚15642.6435546875" proc.memory.rssMB‚ 2720.7265625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚15642.6435546875 disk.sda.in‚0 disk.sda3.out‚ 145.078125 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚15642.6435546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 9503.1796875* disk./.usagePercent‚40.046212534423795 disk.sda3.in‚0 cpu‚26.085861422443344$ memory_percent‚95.45800685882568) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17( proc.memory.availableMB‚ 744.16015625 network.recv‚ 1.267687e+06$ disk./.usageGB‚596.7175064086914 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9503.1796875 disk.nvme0n1.in‚ 9503.1796875 disk.sda1.in‚0 disk.drbd2165.in‚0^)ñ’Þ‚(:‡ ãëÒ¬ô¸“ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚254.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ç:ÞÑß‚(:Æ ãëÒµÌð– disk.sda.out‚ 146.6328125& disk.nvme0n1.out‚15655.44384765625 disk.rbd1.out‚ 11.58203125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2720.7265625) disk./.usagePercent‚40.04596548715325! disk.nvme0n1.in‚ 9504.01171875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15655.44384765625( proc.memory.availableMB‚ 743.91015625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17$ memory_percent‚95.45953273773193 network.sent‚ 8.581439e+06 network.recv‚ 1.270198e+06 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 9504.01171875$ disk./.usageGB‚596.7138252258301 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 146.6328125 disk.drbd2165.out‚0 disk.sda3.out‚ 146.6328125 disk.dm-0.in‚ 9504.01171875# disk.dm-0.out‚15655.44384765625 disk.sda2.out‚0) proc.memory.percent‚16.60599708557129 disk.drbd2165.in‚0 cpu‚26.086646206140962hÍܶ2à- _wandbruntime‚4638  _runtime‚4638U&šúÇáÁ % _timestamp‚1.7843449798537323e+09 _runtime‚4638.547380844 _step‚445 & lr-AdamW/pg1‚0.0003794743388027044 & lr-AdamW/pg2‚0.0003794743388027044 trainer/global_step‚5449PF¦„ÌâÆ _runtime‚4638.547380844 _step‚445 & lr-AdamW/pg1‚0.0003794743388027044 & lr-AdamW/pg2‚0.0003794743388027044 trainer/global_step‚5449 % _timestamp‚1.7843449798537323e+09½ºEü2ã- _wandbruntime‚4638  _runtime‚4638ÖlMá'äÛO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734263296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5756932370432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166918 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827583818932224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5818078252544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827583801892864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚ 4638.54857595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833341092641792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188335743273472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823331067301888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 trainer/global_step‚5449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9548891 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5818173121024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456261 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446284 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5757273709568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11714931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829079551993364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827583818932224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21263562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827583801892864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188335629792768 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496¤¯§QÌåàO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5756932370432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166918 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827583818932224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5818078252544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827583801892864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚ 4638.54857595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833341092641792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188335743273472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823331067301888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 trainer/global_step‚5449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9548891 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5818173121024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456261 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446284 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5757273709568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11714931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829079551993364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827583818932224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21263562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827583801892864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623181 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188335629792768 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734263296¾P(Ž12æ- _wandbruntime‚4639  _runtime‚4639t€íì&çæM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827731059961856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21267328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757944997376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745310 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50465327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823477560934400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346305536 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11716976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50465327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11716695 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5819193308672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚5449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5758286343680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843449804687223e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719070 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5819098447360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629886 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚447 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346305536 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11716976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50465327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11716695 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5819193308672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚5449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5758286343680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843449804687223e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719070 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5819098447360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629886 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚447 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827731059961856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21267328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757944997376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745310 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496¿µ„Š2é- _wandbruntime‚4639  _runtime‚4639ôøдêÿ 0loss/FRt1/assignment_loss‚1.6826491456920818 .loss/FRt1/detection_loss‚0.617416038728276 .loss/FBt1/detection_loss‚0.930932904170318 trainer/global_step‚5449 0loss/SRqqt2/detection_loss‚0.645764069250817 %loss/total_loss‚8.632898739287281 epoch‚13 _runtime‚4639.164642032 _step‚448 3loss/SRqqt2/assignment_loss‚0.46322238688506706 /loss/FRt2/detection_loss‚1.3348133332721324 0loss/SRqqt1/detection_loss‚0.878009284329732 /loss/FBt2/detection_loss‚0.5412374469329558 0loss/FRt2/assignment_loss‚0.9193804503716144 2loss/SRqqt1/assignment_loss‚0.5918300920466237 3loss/FBt1/assignment_loss‚0.0012022301790207994 1loss/FBt2/assignment_loss‚0.02644135742864237 # _timestamp‚1.78434498047134e+09®é ë„ 3loss/SRqqt2/assignment_loss‚0.46322238688506706 /loss/FRt2/detection_loss‚1.3348133332721324 0loss/SRqqt1/detection_loss‚0.878009284329732 /loss/FBt2/detection_loss‚0.5412374469329558 0loss/FRt2/assignment_loss‚0.9193804503716144 2loss/SRqqt1/assignment_loss‚0.5918300920466237 3loss/FBt1/assignment_loss‚0.0012022301790207994 1loss/FBt2/assignment_loss‚0.02644135742864237 # _timestamp‚1.78434498047134e+09 0loss/FRt1/assignment_loss‚1.6826491456920818 .loss/FRt1/detection_loss‚0.617416038728276 .loss/FBt1/detection_loss‚0.930932904170318 trainer/global_step‚5449 0loss/SRqqt2/detection_loss‚0.645764069250817 %loss/total_loss‚8.632898739287281 epoch‚13 _runtime‚4639.164642032 _step‚448À˜Š$ùžì‚(:“ ŸãëÒš†±“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZoŸRÑí‚(:Æ ŸãëÒ½•Ž• proc.cpu.threads‚17 disk.dm-1.out‚ 147.95703125 disk.rbd1.in‚0 disk.rbd1.out‚ 11.58203125$ memory_percent‚95.45984268188477) proc.memory.percent‚16.60599708557129& disk.nvme0n1.out‚15659.82861328125 disk.sda3.in‚0 disk.sda3.out‚ 147.95703125 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 147.95703125! disk.nvme0n1.in‚ 9593.38671875& proc.memory.availableMB‚ 743.859375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 9593.38671875" proc.memory.rssMB‚ 2720.7265625 cpu‚26.08777814646779 network.sent‚ 8.586375e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚15659.82861328125$ disk./.usageGB‚596.7142639160156 disk.drbd2165.out‚0# disk.dm-0.out‚15659.82861328125 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚ 1.273738e+06) disk./.usagePercent‚40.04599492801969 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 9593.38671875 disk.sda2.in‚0ªõ4“î‚(:ˆ ®ãëÒµ„©” gpu.0.gpu‚50.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hd^Gï‚(:¼ ®ãëÒº€À– network.sent‚ 8.64938e+06 network.recvj $€‚ 1.27897e+06 disk.nvme0n1.in‚ 9594.3515625 disk.sda3.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚15670.31298828125 cpu‚26.08865331227835 disk.dm-1.out‚ 148.703125# memory_percent‚95.4594373703003" proc.memory.rssMB‚ 2720.7265625 disk.sda.out‚ 148.703125 disk.rbd1.out‚ 11.64453125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚15670.31298828125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 9594.3515625) proc.memory.percent‚16.60599708557129) disk./.usagePercent‚40.04589764515668$ disk./.usageGB‚596.7128143310547 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 148.703125 disk.rbd1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 743.92578125 disk.dm-0.in‚ 9594.3515625# disk.dm-0.out‚15670.31298828125 disk.rbd0.in‚0zÛ–J2ð- _wandbruntime‚4679  _runtime‚4679LdštÉñà ' lr-AdamW/pg1‚0.00037816690704449413 ' lr-AdamW/pg2‚0.00037816690704449413 trainer/global_step‚5499 % _timestamp‚1.7843450205757432e+09 _runtime‚4679.269389465 _step‚449 ®{óÎòÈ ' lr-AdamW/pg1‚0.00037816690704449413 ' lr-AdamW/pg2‚0.00037816690704449413 trainer/global_step‚5499 % _timestamp‚1.7843450205757432e+09 _runtime‚4679.269389465 _step‚449Á¬2ó- _wandbruntime‚4679  _runtime‚4679§·½Pâ'ôÜO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4679.270795529 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚836455702073872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21454003 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190058283097600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184188989466624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 trainer/global_step‚5499 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21453740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43548518 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807627872256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5807627872256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834945853374464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5869275018752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11817862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190058180102656 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5869190636032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969211392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9636138 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910713 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834945853374464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969211392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9636141 % _timestamp‚1.7843450205769045e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481246720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43548520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836455344484622 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840753839625216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÐfZ~ç'õáO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481246720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43548520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836455344484622 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840753839625216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4679.270795529 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚836455702073872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21454003 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190058283097600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184188989466624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 trainer/global_step‚5499 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21453740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43548518 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807627872256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5807627872256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834945853374464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5869275018752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11817862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190058180102656 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5869190636032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969211392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9636138 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910713 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834945853374464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969211392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9636141 % _timestamp‚1.7843450205769045e+09Â’5¤ß2ö- _wandbruntime‚4679  _runtime‚4679Šªßë&÷åM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11819718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 $ _timestamp‚1.784345021192965e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5800112232218 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚830803111567360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808981845504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21457601 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11819981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5870216811520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43556150 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5870301187072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154Õ²PYøêM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50919779 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50909788 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50909788 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚836603223799578 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475931däg—# N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808981845504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21457601 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11819981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5870216811520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43556150 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5870301187072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11819718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 $ _timestamp‚1.784345021192965e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5800112232218 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚830803111567360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43930120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21644407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 _runtime‚4719.646555714 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5920379513344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843450609528303e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843831494565130 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚848166228230144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843831852154380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚454 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11921019 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11921285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191780907804160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191780806906368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586608640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115402 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355188 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51365165 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664713216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51355188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664713216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976]#‚¬‡áO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115402 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355188 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51365165 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664713216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51355188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664713216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842307921895424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5858323374080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842307904856064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185860446005248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5920461798912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43930120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21644407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 _runtime‚4719.646555714 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5920379513344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843450609528303e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843¨O†; 831494565130 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚848166228230144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843831852154380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚454 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11921019 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11921285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191780907804160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191780806906368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586608640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ÆFgA2ˆ- _wandbruntime‚4720  _runtime‚4720 㤺ì&‰æM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677347328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314840272384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11923043 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5921494146048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11923286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5859336001024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677347328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880086 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4720.260522651 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21648176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719126 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9725133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185893874763776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314481886720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185893856151552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314840272384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚5549 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51374231 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5859336001024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚838128628121600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚838128628121600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5921494146048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11923286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5859336001024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677347328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880086 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4720.260522651 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21648176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719126 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9725133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185893874763776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314481886720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185893856151552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438Ç2vþ 2‹- _wandbruntime‚4720  _runtime‚4720®‰¬ Œ† _runtime‚4720.263063818 0loss/FRt2/assignment_loss‚0.9309259359387384 /loss/FRt1/detection_loss‚0.5987693614933208 epoch‚13 _step‚456 0loss/FBt2/detection_loss‚0.48274147341938783 0loss/FRt1/assignment_loss‚1.7206050625611697 2loss/SRqqt1/assignment_loss‚0.6875565922644644 3loss/FBt1/assignment_loss‚0.0002859608250667457 3loss/FBt2/assignment_loss‚0.0027631674805110216 /loss/FRt2/detection_loss‚1.3963217336017306 1loss/SRqqt1/detection_loss‚0.9481018211917059 % _timestamp‚1.7843450615697591e+09 2loss/SRqqt2/assignment_loss‚0.5026693734472168 $loss/total_loss‚8.50931092654806 trainer/global_step‚5549 1loss/SRqqt2/detection_loss‚0.6693581433527703 /loss/FBt1/detection_loss‚0.5692123009719783ïÑòЋ 0loss/FRt1/assignment_loss‚1.7206050625611697 2loss/SRqqt1/assignment_loss‚0.6875565922644644 3loss/FBt1/assignment_loss‚0.0002859608250667457 3loss/FBt2/assignment_loss‚0.0027631674805110216 /loss/FRt2/detection_loss‚1.3963217336017306 1loss/SRqqt1/detection_loss‚0.9481018211917059 % _timestamp‚1.7843450615697591e+09 2loss/SRqqt2/assignment_loss‚0.5026693734472168 $loss/total_loss‚8.50931092654806 trainer/global_step‚5549 1loss/SRqqt2/detection_loss‚0.6693581433527703 /loss/FBt1/detection_loss‚0.5692123009719783 _runtime‚4720.263063818 0loss/FRt2/assignment_loss‚0.9309259359387384 /loss/FRt1/detection_loss‚0.5987693614933208 epoch‚13 _step‚456 0loss/FBt2/detection_loss‚0.48274147341938783ÈÒÒ)ƒ“Ž‚(:ˆ êãëÒ¿¬§“ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÚ Ì‚(:Á êãëÒΞŽ–# disk./.usageGB‚596.711971282959 disk.sda.out‚ 150.7109375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 497.3515625) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 disk.dm-1.out‚ 150.7109375 disk.sda3.out‚ 150.7109375 disk.rbd1.out‚ 11.83203125 disk.rbd0.in‚0 disk.sda1.out‚0" disk.dm-0.out‚15697.6962890625 network.sent‚ 8.727229e+06 disk.rbd0.out‚0 disk.sda1.in‚0 network.recv‚ 1.293895e+06* disk./.usagePercent‚40.045841067491615 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 9693.38671875% disk.nvme0n1.out‚15697.6962890625 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 9693.38671875 disk.dm-0.in‚ 9693.38671875" proc.memory.rssMB‚ 2720.7265625 cpu‚26.09216672384679 disk.sda3.in‚0' disk.nvme0c0n1.out‚15697.6962890625$ memory_percent‚96.96440696716309 disk.sda2.out‚0ÑoGž‚(:“ ùãëÒ‚ì” gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.412' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.282400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A¾¡o»‘‚(:° ùãëÒ‹…– network.sent‚ 8.79024e+06 network.recv‚ 1.299258e+06 disk.nvme0n1.in‚ 9694.21875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 9694.21875" proc.memory.rssMB‚ 2720.7265625) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 disk.dm-0.in‚ 9694.21875 disk.sda2.in‚0( proc.memory.availableMB‚ 497.61328125 disk.sda3.out‚151.125 disk.dm-1.out‚151.125 disk.sda.in‚0# memory_percent‚96.9628095626831 disk.sda3.in‚0 disk.drbd2165.in‚0 cpu‚26.093135079608246& disk.nvme0n1.out‚15705.15966796875 disk.rbd1.out‚ 11.83203125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚15705.15966796875 disk.sda.out‚151.125* disk./.usagePercent‚40.045873324440926$ disk./.usageGB‚596.7124519348145( disk.nvme0c0n1.out‚15705.15966796875]û°ÿ“’‚(:ˆ ˆäëÒëÕØ” gpu.0.gpu‚66.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚235.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï\uYГ‚(:Å ˆäëÒÍÆ€–# disk.nvme0c0n1.in‚ 9695.18359375 network.sent‚ 8.795177e+06 disk.sda2.in‚0! disk.nvme0n1.in‚ 9695.18359375 disk.sda.out‚ 151.5859375 disk.rbd0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚15715.52978515625) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17& disk.nvme0n1.out‚15715.52978515625( disk./.usagePercent‚40.0459030213149 disk.sda2.out‚0 disk.dm-1.out‚ 151.5859375 disk.sda3.in‚0 disk.rbd1.out‚ 11.89453125 disk.dm-0.in‚ 9695.18359375 cpu‚26.093941629330214$ disk./.usageGB‚596.7128944396973 disk.sda.in‚0( proc.memory.availableMB‚ 497.76171875 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚15715.52978515625 network.recv‚ 1.302394e+06$ memory_percent‚96.96190357208252" proc.memory.rssMB‚ 2720.7265625 disk.sda3.out‚ 151.5859375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0ÇY^2”- _wandbruntime‚4760  _runtime‚4760òlj9Ç•Á ' lr-AdamW/pg2‚0.00037552252633368474 trainer/global_step‚5599 % _timestamp‚1.7843451015867462e+09 _runtime‚ 4760.2803406 _step‚457 ' lr-AdamW/pg1‚0.00037552252633368474f鉭Ì–Æ ' lr-AdamW/pg1‚0.00037552252633368474 ' lr-AdamW/pg2‚0.00037552252633368474 trainer/global_step‚5599 % _timestamp‚1.7843451015867462e+09 _runtime‚ 4760.2803406 _step‚457Éa°À2—- _wandbruntime‚4760  _runtime‚4760Þ°Rð× ˜ÜO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975213568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849669956337664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851207644645638 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12024068 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12023799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360215040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193503433035776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4760.281513394 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9810638 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21834709 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96121358 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187531865319424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5971460527104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849669956337664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5909018875904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843451015877893e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚855579333592064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851208002234888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21834437 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚849669973377024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51819617 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193503325846528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚458 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5971549104128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819617 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154GÜ9Dç'™áO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975213568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849669956337664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851207644645638 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12024068 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12023799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360215040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193503433035776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4760.281513394 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9810638 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21834709 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96121358 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187531865319424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5971460527104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849669956337664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5909018875904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843451015877893e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚855579333592064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851208002234888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21834437 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚849669973377024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51819617 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193503325846528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚458 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5971549104128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819617 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983ÊãDŒj2š- _wandbruntime‚4760  _runtime‚4760Ž Œqì&›æM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12025902 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21838565 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818692 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031502848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843451022020936e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21838285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206034 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5901004656032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5599 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828683 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372849152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚459 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228870144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5972471981056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4760.895938237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12026179 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187565294077952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5910031502848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851355166371232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193537879532544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587255808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476‹â²ÍœëM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828683 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372849152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚459 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228870144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5972471981056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4760.895938237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12026179 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187565294077952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5910031502848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851355166371232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193537879532544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587255808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12025902 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21838565 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818692 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031502848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843451022020936e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21838285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206034 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5901004656032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5599Ë…æp2- _wandbruntime‚4760  _runtime‚4760æ _ ž‡ 0loss/FRt1/assignment_loss‚1.6366112292195796 3loss/SRqqt2/assignment_loss‚0.49002092793473717 % _timestamp‚1.7843451022048702e+09 _step‚460 epoch‚13 0loss/FRt2/assignment_loss‚0.8799398581558299 2loss/SRqqt1/assignment_loss‚0.5499311767296056 4loss/FBt1/assignment_loss‚0.00016258706447160875 /loss/FBt1/detection_loss‚0.5885844927281141 2loss/FBt2/assignment_loss‚0.004670445542377057 1loss/SRqqt1/detection_loss‚0.9274944729101525 /loss/FBt2/detection_loss‚0.5326744130275284 /loss/FRt1/detection_loss‚0.6005517417739373 /loss/FRt2/detection_loss‚1.1911699439605663 1loss/SRqqt2/detection_loss‚0.6162292784857476 %loss/total_loss‚8.018040567532648 trainer/global_step‚5599 _runtime‚4760.898192001ëLÇ&ŸŒ /loss/FBt2/detection_loss‚0.5326744130275284 /loss/FRt1/detection_loss‚0.6005517417739373 /loss/FRt2/detection_loss‚1.1911699439605663 1loss/SRqqt2/detection_loss‚0.6162292784857476 %loss/total_loss‚8.018040567532648 trainer/global_step‚5599 _runtime‚4760.898192001 0loss/FRt1/assignment_loss‚1.6366112292195796 3loss/SRqqt2/assignment_loss‚0.49002092793473717 % _timestamp‚1.7843451022048702e+09 _step‚460 epoch‚13 0loss/FRt2/assignment_loss‚0.8799398581558299 2loss/SRqqt1/assignment_loss‚0.5499311767296056 4loss/FBt1/assignment_loss‚0.00016258706447160875 /loss/FBt1/detection_loss‚0.5885844927281141 2loss/FBt2/assignment_loss‚0.004670445542377057 1loss/SRqqt1/detection_loss‚0.9274944729101525ÌÝî“ ‚(:ˆ —äëÒÃ÷–• gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ ­Ð¡‚(:Å —äëÒ‡«²– disk.dm-1.out‚ 151.9921875 disk.sda2.out‚0$ memory_percent‚96.96526527404785 disk.sda3.out‚ 151.9921875 disk.rbd1.in‚0 disk.rbd1.out‚ 11.89453125' proc.memory.availableMB‚ 497.2109375$ disk./.usageGB‚596.7126197814941# disk.nvme0c0n1.in‚ 9761.32421875( disk.nvme0c0n1.out‚15718.94482421875! disk.nvme0n1.in‚ 9761.32421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 151.9921875 disk.sda2.in‚0) proc.memory.percent‚16.60599708557129 proc.cpu.threads‚17 network.sent‚ 8.800004e+06 network.recv‚ 1.305941e+06 disk.sda3.in‚0# disk.dm-0.out‚15718.94482421875) disk./.usagePercent‚40.04588458877244 disk.dm-1.in‚0 disk.dm-0.in‚ 9761.32421875& disk.nvme0n1.out‚15718.94482421875 cpu‚26.100989217587824 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2720.7265625 disk.rbd0.out‚0 disk.sda1.out‚0d§¬2¢- _wandbruntime‚4778  _runtime‚4778ˆWÏ9ç=£á{ % _timestamp‚1.7843451201391847e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2902490198612213 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800741493701935 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980595707893372 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.662853479385376 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8067134022712708 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6637582182884216 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49593472480773926 6"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7594034075737 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7280093431472778 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2163018435239792 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6891322135925293 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_2‚0.7816129326820374 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7537379264831543 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3880404829978943 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7517467141151428 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6695048213005066 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8783824443817139 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488824009895325 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985962986946106 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2236364483833313 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7480092644691467 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7457209825515747 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4816860854625702 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025789575651288033 /particle/accuracy_4_of_4‚0.9423011541366577 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6396758556365967 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04345400631427765 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3934224545955658 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 5"Purity/*FRt0SRqqt1FBt/event_purity‚0.894287109375 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028736356645822525 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02904460020363331 'particle/f_score‚0.6391555070877075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22151032090187073 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3188883066177368 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5791075825691223 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8051965832710266 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08423139154911041 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.035993337631225586 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871965050697327 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20243822038173676 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05202540382742882 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4867483675479889 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925674200057983 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8635865449905396 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6991685032844543 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 *jet/accuracy_2_of_3‚0.5732316970825195 .particle/accuracy_3_of_4‚0.997619092464447 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚4778.833404237 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11942882835865021 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8541606664657593 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2701546847820282 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09107500314712524 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4762727618217468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972350001335144 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7564994096755981 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23354943096637726 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958937764167786 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2800741493701935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2778197228908539 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39403942227363586 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19848789274692535 epoch‚13 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938545227050781 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8843348622322083 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 )loss/val_total_loss‚8.119396918967416 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7516501545906067 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7038005590438843 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23579688370227814 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9772451519966125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6008020043373108 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.849144458770752 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6652617454528809 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7733100652694702 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7788177728652954 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08311391621828079 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3520447015762329 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6670766472816467 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025789575651288033 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941346049308777 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6860655546188354 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29147693514823914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 _step‚461 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8513228297233582 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865330219268799 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8070338368415833 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7130452990531921 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.587733268737793 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6602837443351746 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8728328943252563 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10106359422206879 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020474672317505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2512665092945099 +particle/specificity‚0.9112205505371094 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9940884113311768 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_3‚0.9153884053230286 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7350485920906067 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.821252167224884 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04122909903526306 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7092215418815613 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2209016978740692 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48245370388031006 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_3‚0.9994649291038513 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995287299156189 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2898336946964264 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22042140364646912 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6461564302444458 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21531841158866882 +jet/accuracy_2_of_2‚0.15698334574699402 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6637582182884216 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6961859464645386 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21214458346366882 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10233704000711441 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06901319324970245 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6614266633987427 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 +jet/accuracy_3_of_3‚0.07283355295658112 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02830863557755947 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6211196780204773 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029654797166585922 trainer/global_step‚5599 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949563145637512 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6547726392745972 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8728328943252563 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3523603081703186 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33536484837532043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7448155283927917 +particle/sensitivity‚0.6221612691879272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21996177732944489 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.271041601896286 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4209398031234741 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.295698881149292 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.876152753829956 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5208040475845337 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6115012168884277 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128„~RœÐ$¤æ{ & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2778197228908539 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39403942227363586 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19848789274692535 epoch‚13 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938545227050781 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8843348622322083 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 )loss/val_total_loss‚8.119396918967416 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7516501545906067 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7038005590438843 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23579688370227814 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9772451519966125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6008020043373108 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.849144458770752 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6652617454528809 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7733100652694702 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7788177728652954 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08311391621828079 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3520447015762329 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6670766472816467 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025789575651288033 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941346049308777 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6860655546188354 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29147693514823914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 _step‚461 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8513228297233582 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865330219268799 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8070338368415833 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7130452990531921 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.587733268737793 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6602837443351746 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8728328943252563 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10106359422206879 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7020474672317505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2512665092945099 +particle/specificity‚0.9112205505371094 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9940884113311768 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_3‚0.9153884053230286 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7350485920906067 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.821252167224884 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04122909903526306 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7092215418815613 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2209016978740692 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48245370388031006 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_3‚0.9994649291038513 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995287299156189 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2898336946964264 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22042140364646912 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6461564302444458 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21531841158866882 +jet/accuracy_2_of_2‚0.15698334574699402 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6637582182884216 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6961859464645386 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21214458346366882 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10233704000711441 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06901319324970245 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6614266633987427 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 +jet/accuracy_3_of_3‚0.07283355295658112 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02830863557755947 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6211196780204773 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029654797166585922 trainer/global_step‚5599 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949563145637512 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6547726392745972 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8728328943252563 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3523603081703186 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33536484837532043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7448155283927917 +particle/sensitivity‚0.6221612691879272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21996177732944489 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.271041601896286 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4209398031234741 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.295698881149292 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.876152753829956 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5208040475845337 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1Sœé6ORqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6115012168884277 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 % _timestamp‚1.7843451201391847e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2902490198612213 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800741493701935 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980595707893372 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.662853479385376 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8067134022712708 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6637582182884216 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49593472480773926 6"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7594034075737 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7280093431472778 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2163018435239792 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6891322135925293 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_2‚0.7816129326820374 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974368810653687 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7537379264831543 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3880404829978943 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7517467141151428 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6695048213005066 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8783824443817139 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488824009895325 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985962986946106 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2236364483833313 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7480092644691467 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7457209825515747 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4816860854625702 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025789575651288033 /particle/accuracy_4_of_4‚0.9423011541366577 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6396758556365967 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04345400631427765 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3934224545955658 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 5"Purity/*FRt0SRqqt1FBt/event_purity‚0.894287109375 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028736356645822525 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02904460020363331 'particle/f_score‚0.6391555070877075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22151032090187073 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3188883066177368 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5791075825691223 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8051965832710266 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08423139154911041 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.035993337631225586 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871965050697327 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20243822038173676 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05202540382742882 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4867483675479889 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925674200057983 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8635865449905396 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6991685032844543 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 *jet/accuracy_2_of_3‚0.5732316970825195 .particle/accuracy_3_of_4‚0.997619092464447 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚4778.833404237 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11942882835865021 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8541606664657593 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2701546847820282 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09107500314712524 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4762727618217468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972350001335144 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7564994096755981 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23354943096637726 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958937764167786 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2800741493701935ÍG6“¥‚(:ˆ ¦äëÒ¯˜Ü” gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚238.668' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@FºÅ¦‚(:º ¦äëÒÁ——) disk./.usagePercent‚40.04596036700257 disk.drbd2165.out‚0$ memory_percent‚95.47474384307861 network.sent‚ 8.862928e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 9762.15625) proc.memory.percent‚16.60630702972412# disk.dm-0.out‚15973.85302734375 disk.sda2.in‚0 disk.nvme0n1.in‚ 9762.15625( proc.memory.availableMB‚ 741.41796875 cpu‚26.10715738995228 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 152.484375& disk.nvme0n1.out‚15973.85302734375# proc.memory.rssMB‚ 2720.77734375$ disk./.usageGB‚596.7137489318848 disk.dm-1.in‚0 disk.dm-0.in‚ 9762.15625 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.95703125 disk.dm-1.out‚ 152.484375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚15973.85302734375 disk.sda2.out‚0 disk.sda3.out‚ 152.484375 proc.cpu.threads‚17 network.recv‚ 1.311114e+06ÎY¶÷§‚(:’ µäëÒÑÓ“ gpu.0.gpu‚47.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚280.287' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.05739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C¼Uvè‚(:¸ µäëÒØÏõ• disk.dm-0.in‚ 9763.12109375 disk.sda.in‚0 network.sent‚ 8.920329e+06 disk.sda3.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.60630702972412& disk.nvme0c0n1.out‚16228.126953125$ disk./.usageGB‚596.7138137817383 disk.sda2.in‚0 disk.sda1.out‚0# memory_percent‚95.4747200012207 disk.rbd0.in‚0! disk.nvme0n1.in‚ 9763.12109375 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 9763.12109375& proc.memory.availableMB‚ 741.421875 disk.sda2.out‚0 disk.rbd1.out‚ 11.95703125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 152.78125 disk.drbd2165.in‚0 cpu‚26.107511874119933 disk.rbd0.out‚0 network.recv‚ 1.316204e+06) disk./.usagePercent‚40.04596471913065 disk.sda3.out‚ 152.78125# proc.memory.rssMB‚ 2720.77734375! disk.dm-0.out‚16228.126953125 proc.cpu.threads‚17$ disk.nvme0n1.out‚16228.126953125 disk.sda.out‚ 152.78125…ðd_ž©‚(:“ ÄäëÒ¾„ü” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fDI&ɪ‚(:¾ ÄäëÒ—ç’– disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚16233.447265625 disk.rbd1.in‚0( proc.memory.availableMB‚ 741.36328125 disk.dm-1.in‚0 disk.sda.out‚ 153.046875 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 8.925361e+06 disk.sda3.out‚ 153.046875* disk./.usagePercent‚40.045970863311474$ disk./.usageGB‚596.7139053344727 disk.dm-0.in‚ 9763.12109375! disk.nvme0n1.in‚ 9763.12109375 disk.rbd0.in‚0 cpu‚26.10800223354548 proc.cpu.threads‚17 network.recv‚ 1.319389e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 153.046875# disk.nvme0c0n1.in‚ 9763.12109375! disk.dm-0.out‚16233.447265625 disk.sda.in‚0 disk.rbd1.out‚ 12.01953125$ disk.nvme0n1.out‚16233.447265625$ memory_percent‚95.47507762908936# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 disk.sda2.in‚0ô—662«- _wandbruntime‚4820  _runtime‚4820Ì®otŬ¿ # _timestamp‚1.78434516204487e+09 _runtime‚4820.738380472 _step‚462 & lr-AdamW/pg1‚0.0003741857472247173 & lr-AdamW/pg2‚0.0003741857472247173 trainer/global_step‚5649ؼßFÊ­Ä & lr-AdamW/pg1‚0.0003741857472247173 & lr-AdamW/pg2‚0.0003741857472247173 trainer/global_step‚5649 # _timestamp‚1.78434516204487e+09 _runtime‚4820.738380472 _step‚462Îhé2®- _wandbruntime‚4820  _runtime‚4820aZ‘â'¯ÜO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843451620459733e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380433920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚4820.739712777 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6041005212672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6040920829952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5968670364342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚864939380433920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12159863 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195260239170560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334419 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148403 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447585792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22060004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22060274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224³sŸuç'°áO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6040920829952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5968670364342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚864939380433920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12159863 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195260239170560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334419 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148403 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447585792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22060004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22060274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843451620459733e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380433920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚4820.739712777 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6041005212672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Ï uÏØ2±- _wandbruntime‚4821  _runtime‚4821ŸŠç–n²åM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12161622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDevª6}"iceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460212736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189252662716416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195294703635456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚860657699391054 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22063775 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6042040919040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859108815499264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9902153 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859108832538624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4821.352102239 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52343471 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165087 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9902156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69695488 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195294615327744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚860658056980416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5978801559040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12161884 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52353462 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360ÚHØð&³êM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6042040919040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859108815499264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9902153 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859108832538624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4821.352102239 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52343471 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165087 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9902156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69695488 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195294615327744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚860658056980416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5978801559040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12161884 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52353462 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12161622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460212736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189252662716416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195294703635456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚860657699391054 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22063775 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616мìd2´- _wandbruntime‚4821  _runtime‚4821}¼n µ„ /loss/FRt2/detection_loss‚1.2316067539043083 1loss/SRqqt2/detection_loss‚0.6507495506410093 /loss/FBt1/detection_loss‚0.6141308607119653 /loss/FBt2/detection_loss‚0.3837728639136649 _runtime‚4821.355543155 _step‚465 0loss/FRt1/assignment_loss‚1.7109339116774884 0loss/FRt2/assignment_loss‚0.9090250464530311 2loss/FBt1/assignment_loss‚0.007466622289508291 epoch‚14 /loss/FRt1/detection_loss‚0.5562646405013287 %loss/total_loss‚8.035075606943103 trainer/global_step‚5649 % _timestamp‚1.7843451626622405e+09 2loss/SRqqt1/assignment_loss‚0.6061581843547609 2loss/SRqqt2/assignment_loss‚0.4990458640350759 2loss/FBt2/assignment_loss‚0.035972380789081965 1loss/SRqqt1/detection_loss‚0.8299489276718807¢ÙT¶‰ /loss/FRt2/detection_loss‚1.2316067539043083 1loss/SRqqt2/detection_loss‚0.6507495506410093 /loss/FBt1/detection_loss‚0.6141308607119653 /loss/FBt2/detection_loss‚0.3837728639136649 _runtime‚4821.355543155 _step‚465 0loss/FRt1/assignment_loss‚1.7109339116774884 0loss/FRt2/assignment_loss‚0.9090250464530311 2loss/FBt1/assignment_loss‚0.007466622289508291 epoch‚14 /loss/FRt1/detection_loss‚0.5562646405013287 %loss/total_loss‚8.035075606943103 trainer/global_step‚5649 % _timestamp‚1.7843451626622405e+09 2loss/SRqqt1/assignment_loss‚0.6061581843547609 2loss/SRqqt2/assignment_loss‚0.4990458640350759 2loss/FBt2/assignment_loss‚0.035972380789081965 1loss/SRqqt1/detection_loss‚0.8299489276718807Ñ^Χ×·‚(:’ ÓäëÒËÚ” gpu.0.gpu‚46.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ÌY^ȸ‚(:½ ÓäëÒöù”—$ disk./.usageGB‚596.7138328552246 disk.sda1.in‚0 disk.rbd1.out‚ 12.01953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚ 1.322716e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 153.5078125 disk.sda1.out‚0 disk.dm-0.in‚ 9856.63671875! disk.nvme0n1.in‚ 9856.63671875$ memory_percent‚95.47672271728516% proc.memory.availableMB‚ 741.09375* disk./.usagePercent‚40.045965999168324 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 153.5078125& disk.nvme0c0n1.out‚16240.205078125! disk.dm-0.out‚16240.205078125$ disk.nvme0n1.out‚16240.205078125 disk.sda.in‚0 disk.rbd1.in‚0) proc.memory.percent‚16.60630702972412# disk.nvme0c0n1.in‚ 9856.63671875# proc.memory.rssMB‚ 2720.77734375 cpu‚26.108892358533286 network.sent‚ 8.9303e+06 disk.sda.out‚ 153.5078125¸±¯Tž¹‚(:“ âäëÒ”òß• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅF³‡Ðº‚(:Å âäëÒîú¹— network.sent‚ 8.995474e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 9857.16015625 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚95.47569751739502) proc.memory.percent‚16.60630702972412 network.recv‚ 1.32766e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 9857.16015625 proc.cpu.threads‚17) disk./.usagePercent‚40.04597598346216 disk.sda2.in‚0% disk.nvme0n1.out‚16244.1240234375 disk.sda3.out‚ 153.67578125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 153.67578125 disk.rbd1.out‚ 12.08203125 disk.sda1.in‚0' disk.nvme0c0n1.out‚16244.1240234375" disk.dm-0.out‚16244.1240234375 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚26.109784886641954# disk./.usageGB‚596.713981628418 disk.sda.in‚0! disk.nvme0n1.in‚ 9857.16015625 disk.rbd0.in‚0 disk.sda.out‚ 153.67578125( proc.memory.availableMB‚ 741.26171875# proc.memory.rssMB‚ 2720.77734375Ùﻂ(:’ ñäëÒŽ±È“ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç8–„ͼ‚(: ñäëÒæ€ä•! disk.nvme0n1.in‚ 9857.63671875 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 154.046875$ memory_percent‚95.47419548034668 disk.dm-0.in‚ 9857.63671875 disk.sda.out‚ 154.046875) proc.memory.percent‚16.60630702972412 cpu‚26.11064291204157 network.sent‚ 8.998396e+06) disk./.usagePercent‚40.04602948903682 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.dm-0.out‚16251.47119140625 disk.sda3.in‚0$ disk./.usageGB‚596.7147789001465 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 9857.63671875( disk.nvme0c0n1.out‚16251.47119140625& disk.nvme0n1.out‚16251.47119140625 disk.rbd1.out‚ 12.08203125 disk.sda1.in‚0# proc.memory.rssMB‚ 2720.77734375 network.recv‚ 1.331065e+06 disk.sda3.out‚ 154.046875' proc.memory.availableMB‚ 741.5078125 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0oN_2½- _wandbruntime‚4861  _runtime‚4861úMt—ž¿ & lr-AdamW/pg1‚0.0003728393565508578 & lr-AdamW/pg2‚0.0003728393565508578 trainer/global_step‚5699 # _timestamp‚1.78434520253175e+09 _runtime‚4861.225339417 _step‚466°!ÌFÊ¿Ä & lr-AdamW/pg1‚0.0003728393565508578 & lr-AdamW/pg2‚0.0003728393565508578 trainer/global_step‚5699 # _timestamp‚1.78434520253175e+09 _runtime‚4861.225339417 _step‚466Ò}‚ß\2À- _wandbruntime‚4861  _runtime‚4861[¾'ž¨ÁÚO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52798848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6028143087616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872352127417344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994434 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12263153 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6092312838656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45195584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6028484426752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22251097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45195586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867886327745968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769038848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861867381444608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127417344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚527888™@O•8 71 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196982966110720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6019286851250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚5699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45195586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 $ _timestamp‚1.784345202532811e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12263436 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22250811 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552ø0Éå'ÂßO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52798848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6028143087616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872352127417344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994434 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12263153 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6092312838656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45195584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6028484426752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22251097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45195586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867886327745968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769038848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861867381444608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127417344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52788871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196982966110720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6019286851250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚5699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45195586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 $ _timestamp‚1.784345202532811e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12263436 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22250811 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ó õF2Ã- _wandbruntime‚4861  _runtime‚4861˜AÐì&ÄæM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381081088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45203218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197017521910784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22254522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚866470866980864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6019957355082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52807914 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6093326406656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872500022695424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9989406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190924100642816 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6020297905084 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6093421267968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868033849471562 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4861.839211511 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6029155714560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52797923 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6029497060864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12265119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12265401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÈVQF¢&ÅëM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872500022695424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9989406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190924100642816 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6020297905084 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6093421267968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868033849471562 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4861.839211511 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6029155714560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52797923 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6029497060864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12265119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12265401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381081088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45203218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197017521910784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22254522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚866470866980864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6019957355082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52807914 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6093326406656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStats‡%φOMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924Ô•líÉ2Æ- _wandbruntime‚4861  _runtime‚4861ÜÔ^Ç‚ /loss/FRt1/detection_loss‚0.5868681414520037 1loss/SRqqt2/detection_loss‚0.6093747504671067 2loss/SRqqt1/assignment_loss‚0.5651486782276081 /loss/FBt1/detection_loss‚0.4566048108196507 epoch‚14 $ _timestamp‚1.784345203149133e+09 %loss/total_loss‚7.949583246543774 .loss/FRt1/assignment_loss‚1.68450426249019 2loss/SRqqt2/assignment_loss‚0.5608566486512763 3loss/FBt1/assignment_loss‚0.0002178564221440027 2loss/FBt2/assignment_loss‚0.016413527215815543 /loss/FRt2/detection_loss‚1.1866785926970957 trainer/global_step‚5699 _step‚469 1loss/SRqqt1/detection_loss‚0.9312258801221567 /loss/FBt2/detection_loss‚0.4195478854690419 _runtime‚4861.842454733 0loss/FRt2/assignment_loss‚0.9321422125096848JFŠq ȇ /loss/FRt2/detection_loss‚1.1866785926970957 trainer/global_step‚5699 _step‚469 1loss/SRqqt1/detection_loss‚0.9312258801221567 /loss/FBt2/detection_loss‚0.4195478854690419 _runtime‚4861.842454733 0loss/FRt2/assignment_loss‚0.9321422125096848 /loss/FRt1/detection_loss‚0.5868681414520037 1loss/SRqqt2/detection_loss‚0.6093747504671067 2loss/SRqqt1/assignment_loss‚0.5651486782276081 /loss/FBt1/detection_loss‚0.4566048108196507 epoch‚14 $ _timestamp‚1.784345203149133e+09 %loss/total_loss‚7.949583246543774 .loss/FRt1/assignment_loss‚1.68450426249019 2loss/SRqqt2/assignment_loss‚0.5608566486512763 3loss/FBt1/assignment_loss‚0.0002178564221440027 2loss/FBt2/assignment_loss‚0.016413527215815543Õµ[v²’É‚(:‡ €åëұЪ– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚242.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.427 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿#ÛíÌÊ‚(:Á €åëÒ”Íë—$ disk./.usageGB‚596.7146949768066 disk.nvme0c0n1.in‚ 9858.46875 disk.rbd1.out‚ 12.14453125 disk.dm-1.out‚ 154.37890625( proc.memory.availableMB‚ 741.48828125# proc.memory.rssMB‚ 2720.77734375 cpu‚26.111556196769573 proc.cpu.threads‚17) disk./.usagePercent‚40.04602385687106# disk.dm-0.out‚16261.49658203125 disk.nvme0n1.in‚ 9858.46875& disk.nvme0n1.out‚16261.49658203125 disk.sda1.out‚0) proc.memory.percent‚16.60630702972412 disk.sda3.out‚ 154.37890625 disk.rbd1.in‚0 network.recv‚ 1.334132e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 network.sent‚ 9.016394e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 154.37890625 disk.sda2.out‚0 disk.drbd2165.out‚0$ memory_percent‚95.47431468963623( disk.nvme0c0n1.out‚16261.49658203125 disk.dm-0.in‚ 9858.46875 disk.dm-1.in‚0¢íž“Ë‚(:ˆ åëÒáÀ–” gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉiJÉÌ‚(:¾ åëÒ󼚕 disk.rbd1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 disk.rbd1.out‚ 12.14453125 disk.rbd0.in‚0 disk.sda3.out‚ 154.92578125 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 9973.52734375% disk.nvme0c0n1.out‚16263.97265625 disk.sda.out‚ 154.92578125 proc.cpu.threads‚17 network.sent‚ 9.0664e+06 disk.rbd0.out‚0 disk.sda3.in‚0) disk./.usagePercent‚40.04605995393339 disk.sda1.in‚0 disk.dm-1.out‚ 154.92578125 disk.sda.in‚0$ memory_percent‚95.47553062438965' proc.memory.availableMB‚ 741.2890625$ disk./.usageGB‚596.7152328491211 disk.dm-0.in‚ 9973.52734375 disk.sda2.out‚0 network.recv‚ 1.339569e+06 disk.sda2.in‚0! disk.nvme0n1.in‚ 9973.52734375 cpu‚26.112109157104804 disk.drbd2165.in‚0 disk.dm-0.out‚16263.97265625# disk.nvme0n1.out‚16263.97265625HÚ”2Í- _wandbruntime‚4901  _runtime‚4901ˆ|ÄaÇÎÁ _runtime‚4901.542542554 _step‚470 & lr-AdamW/pg1‚0.0003714834407882633 & lr-AdamW/pg2‚0.0003714834407882633 trainer/global_step‚5749 % _timestamp‚1.7843452428488355e+09Á‹ÌÏÆ & lr-AdamW/pg2‚0.0003714834407882633 trainer/global_step‚5749 % _timestamp‚1.7843452428488355e+09 _runtime‚4901.542542554 _step‚470 & lr-AdamW/pg1‚0.0003714834407882633Ö·òír2Ð- _wandbruntime‚4901  _runtime‚4901ñ’ÂÁä'ÑÞO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12366519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198705529497600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6143438299136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843452428500948e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10074908 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820507 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚873685677432832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873685694472192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22441169 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875262477826476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53253300 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚873685677432832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12366261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243323 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198705647172608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4901.543801311 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6078838589440°kßé'ÒãO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6143438299136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843452428500948e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10074908 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820507 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚873685677432832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873685694472192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22441169 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875262477826476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53253300 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚873685677432832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12366261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243323 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198705647172608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4901.543801311 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6078838589440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12366519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198705529497600׳™Ô2Ó- _wandbruntime‚4902  _runtime‚4902u%JÔèM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6070914391992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚5749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198740095885312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22444962 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22445235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6144557316096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843452434645705e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879913128064512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4902.158474025 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _step‚472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656Ê!{ó&ÕíM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22445235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6144557316096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843452434645705e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879913128064512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4902.158474025 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _step‚472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6070914391992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚5749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198740095885312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22444962 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376Ø/Ñ2Ö- _wandbruntime‚4902  _runtime‚4902~¨% ׃ .loss/FRt1/detection_loss‚0.617421372300572 1loss/SRqqt2/detection_loss‚0.6685796797659745 /loss/FBt1/detection_loss‚0.8269399458302125 trainer/global_step‚5749 0loss/FRt1/assignment_loss‚1.6093803257795247 0loss/FRt2/assignment_loss‚0.9210548172150307 2loss/SRqqt2/assignment_loss‚0.5022191691687764 /loss/FRt2/detection_loss‚1.1708725064655983 _runtime‚4902.161370072 2loss/SRqqt1/assignment_loss‚0.6769886708707723 2loss/FBt1/assignment_loss‚0.000538098150333931 2loss/FBt2/assignment_loss‚0.013995660322783475 1loss/SRqqt1/detection_loss‚0.9484405690466597 %loss/total_loss‚8.647735196740523 % _timestamp‚1.7843452434680753e+09 _step‚473 /loss/FBt2/detection_loss‚0.6913043818242863 epoch‚14ž9!¦Øˆ 2loss/SRqqt1/assignment_loss‚0.6769886708707723 2loss/FBt1/assignment_loss‚0.000538098150333931 2loss/FBt2/assignment_loss‚0.013995660322783475 1loss/SRqqt1/detection_loss‚0.9484405690466597 %loss/total_loss‚8.647735196740523 % _timestamp‚1.7843452434680753e+09 _step‚473 /loss/FBt2/detection_loss‚0.6913043818242863 epoch‚14 .loss/FRt1/detection_loss‚0.617421372300572 1loss/SRqqt2/detection_loss‚0.6685796797659745 /loss/FBt1/detection_loss‚0.8269399458302125 trainer/global_step‚5749 0loss/FRt1/assignment_loss‚1.6093803257795247 0loss/FRt2/assignment_loss‚0.9210548172150307 2loss/SRqqt2/assignment_loss‚0.5022191691687764 /loss/FRt2/detection_loss‚1.1708725064655983 _runtime‚4902.161370072Ù5c1!“Ù‚(:ˆ žåëÒÅÓŒ– gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JXõÇÚ‚(:¼ žåëÒ󡨘 disk.dm-1.out‚ 155.421875 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚95.47889232635498 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 12.20703125# disk./.usageGB‚596.715274810791" disk.dm-0.out‚16272.6962890625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 9974.4921875 network.recv‚ 1.342656e+06 disk.nvme0n1.in‚ 9974.4921875% disk.nvme0n1.out‚16272.6962890625 cpu‚26.11300117938603 disk.sda.out‚ 155.421875# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412' disk.nvme0c0n1.out‚16272.6962890625 network.sent‚ 9.071341e+06) disk./.usagePercent‚40.04606277001626 disk.dm-0.in‚ 9974.4921875 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 155.421875( proc.memory.availableMB‚ 740.73828125sU“Û‚(:ˆ ­åëÒŒ¿÷“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áô4¡ÓÜ‚(:È ­åëÒÑþš– disk.rbd0.in‚0* disk./.usagePercent‚40.046418364481326 disk.dm-1.in‚0 disk.dm-1.out‚ 155.91796875( disk.nvme0c0n1.out‚16280.29833984375# disk./.usageGB‚596.720573425293 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 9975.32421875 disk.sda3.out‚ 155.91796875 network.recv‚ 1.347705e+06$ memory_percent‚95.47934532165527' proc.memory.availableMB‚ 740.6640625) proc.memory.percent‚16.60630702972412 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 9975.32421875 disk.sda2.out‚0 disk.rbd1.out‚ 12.20703125 cpu‚26.11364135501426# disk.dm-0.out‚16280.29833984375& disk.nvme0n1.out‚16280.29833984375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.out‚ 155.91796875 network.sent‚ 9.134364e+06# proc.memory.rssMB‚ 2720.77734375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 9975.32421875 žµ@Ý‚(:’ ¼åëÒ€Þ†• gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦“fËÐÞ‚(:Å ¼åëÒÏ…â• disk.drbd2165.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚16298.2724609375 disk.dm-0.in‚ 9975.32421875 disk.sda3.in‚0 cpu‚26.114360383957266 network.recv‚ 1.351102e+06$ disk./.usageGB‚596.7199630737305 disk.dm-1.in‚0 disk.sda.out‚ 157.83984375 disk.rbd1.out‚ 12.26953125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412# disk.nvme0c0n1.in‚ 9975.32421875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 9.139402e+06 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚16298.2724609375! disk.nvme0n1.in‚ 9975.32421875 disk.sda2.in‚0# memory_percent‚95.4801082611084 disk.rbd0.out‚0" disk.dm-0.out‚16298.2724609375 disk.sda.in‚0 disk.sda3.out‚ 157.83984375' proc.memory.availableMB‚ 740.5390625) disk./.usagePercent‚40.04637740327585 disk.sda1.out‚0 disk.dm-1.out‚ 157.83984375ƒU‰2ß- _wandbruntime‚4942  _runtime‚4942ÿ”~>ÇàÁ % _timestamp‚1.7843452833757663e+09 _runtime‚4942.069397043 _step‚474 & lr-AdamW/pg1‚0.0003701180870248695 & lr-AdamW/pg2‚0.0003701180870248695 trainer/global_step‚5799%°^ÊÌáÆ & lr-AdamW/pg1‚0.0003701180870248695 & lr-AdamW/pg2‚0.0003701180870248695 trainer/global_step‚5799 % _timestamp‚1.7843452833757663e+09 _runtime‚4942.069397043 _step‚474ÚÙ¨…2â- _wandbruntime‚4942  _runtime‚4942ÿÖKiä'ãÞO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534091264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53707752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496234 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697775 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875430400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12468708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875430400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200428056447488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12468972 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22631133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200428169928192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263005696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 230251765763Ÿ‚ ¡ äãO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚475 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666538 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194233529124864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6194527322624 % _timestamp‚1.7843452833769746e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚887177621384192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceSta->ubHtsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200428169928192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263005696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534091264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53707752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496234 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697775 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875430400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12468708 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875430400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200428056447488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12468972 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22631133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12470992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22634620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6195538277376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚5799 _runtime‚4942.684686379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169Ïðô&çîM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221940 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12470992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22634620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6195538277376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚5799 _runtime‚4942.684686379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Ü  Žá2è- _wandbruntime‚4942  _runtime‚4942YÿvÍéÿ .loss/FRt2/detection_loss‚1.279912748899636 1loss/SRqqt2/detection_loss‚0.6231582995828407 .loss/FBt2/detection_loss‚0.561470118159148 epoch‚14 0loss/FRt1/assignment_loss‚1.6932643365575537 0loss/FRt2/assignment_loss‚0.8361735701325848 1loss/SRqqt1/detection_loss‚0.9979071564918017 /loss/FBt1/detection_loss‚0.6936391480267048 /loss/FRt1/detection_loss‚0.5740832607770344 _runtime‚4942.687659162 _step‚477 1loss/SRqqt2/assignment_loss‚0.525483491674852 1loss/FBt1/assignment_loss‚0.14398078110648307 $loss/total_loss‚8.55885405016253 trainer/global_step‚5799 % _timestamp‚1.7843452839943528e+09 2loss/SRqqt1/assignment_loss‚0.6152869240600908 2loss/FBt2/assignment_loss‚0.014494214693799707éÎlÎ ê„ 2loss/FBt2/assignment_loss‚0.014494214693799707 .loss/FRt2/detection_loss‚1.279912748899636 1loss/SRqqt2/detection_loss‚0.6231582995828407 .loss/FBt2/detection_loss‚0.561470118159148 epoch‚14 0loss/FRt1/assignment_loss‚1.6932643365575537 0loss/FRt2/assignment_loss‚0.8361735701325848 1loss/SRqqt1/detection_loss‚0.9979071564918017 /loss/FBt1/detection_loss‚0.6936391480267048 /loss/FRt1/detection_loss‚0.5740832607770344 _runtime‚4942.687659162 _step‚477 1loss/SRqqt2/assignment_loss‚0.525483491674852 1loss/FBt1/assignment_loss‚0.14398078110648307 $loss/total_loss‚8.55885405016253 trainer/global_step‚5799 % _timestamp‚1.7843452839943528e+09 2loss/SRqqt1/assignment_loss‚0.6152869240600908ݲ–ë‚(:’ ËåëÒÓÑÞ• gpu.0.gpu‚91.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.936' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.58719999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hP#bÑì‚(:Æ ËåëÒ¶‹ì— disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚95.48165798187256( proc.memory.availableMB‚ 740.28515625) proc.memory.percent‚16.60630702972412 disk.nvme0n1.in‚ 10078.546875 network.recv‚ 1.354458e+06* disk./.usagePercent‚40.046375611223105$ disk./.usageGB‚596.7199363708496 disk.rbd0.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 10078.546875( disk.nvme0c0n1.out‚16306.77587890625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 158.35546875 disk.sda3.out‚ 158.35546875& disk.nvme0n1.out‚16306.77587890625 proc.cpu.threads‚17# disk.dm-0.out‚16306.77587890625 disk.sda3.in‚0 network.sent‚ 9.144282e+06 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 12.26953125 disk.dm-0.in‚ 10078.546875 disk.sda2.out‚0 disk.dm-1.out‚ 158.35546875# proc.memory.rssMB‚ 2720.77734375 cpu‚26.1148467694949ܺ“í‚(:ˆ ÚåëÒ¦»â“ gpu.0.gpu‚87.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚149.493' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IãdrÎî‚(:à ÚåëÒž˜– disk.dm-0.out‚16317.61328125 disk.sda.out‚ 158.78515625 disk.rbd0.out‚0 disk.drbd2165.out‚0$ memory_percent‚95.48275470733643 cpu‚26.11557111005919 disk.dm-0.in‚10079.37890625" disk.nvme0n1.in‚10079.37890625# disk.nvme0n1.out‚16317.61328125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚10079.37890625 disk.rbd1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.359412e+06 disk.dm-1.out‚ 158.78515625# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 network.sent‚ 9.207202e+06( proc.memory.availableMB‚ 740.10546875) disk./.usagePercent‚40.04637919532858$ disk./.usageGB‚596.7199897766113 disk.sda3.out‚ 158.78515625 disk.sda1.out‚0% disk.nvme0c0n1.out‚16317.61328125 disk.sda2.out‚0 disk.rbd1.out‚ 12.33203125u^uÞï‚(:“ éåëÒ£ªó” gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=™x¿Òð‚(:Ç éåëÒÅ“– disk.sda3.out‚ 158.953125 disk.rbd1.in‚0 disk.dm-0.in‚10079.37890625& disk.nvme0n1.out‚16321.50341796875 disk.sda3.in‚0 network.sent‚ 9.212275e+06$ disk.nvme0c0n1.in‚10079.37890625 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 158.953125 proc.cpu.threads‚17 network.recv‚ 1.362928e+06* disk./.usagePercent‚40.046348218416945$ disk./.usageGB‚596.7195281982422 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 158.953125 disk.drbd2165.out‚0 disk.rbd1.out‚ 12.33203125 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚16321.50341796875 disk.sda.in‚0" disk.nvme0n1.in‚10079.37890625$ memory_percent‚96.98207378387451# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.11632354130863( disk.nvme0c0n1.out‚16321.50341796875( proc.memory.availableMB‚ 494.45703125ü0Ì2ñ- _wandbruntime‚4982  _runtime‚4982¿ Éˆòà ' lr-AdamW/pg1‚0.00036874338295479726 ' lr-AdamW/pg2‚0.00036874338295479726 trainer/global_step‚5849 % _timestamp‚1ÄšçA.7843453240116968e+09 _runtime‚4982.705300613 _step‚4785*}ÎóÈ ' lr-AdamW/pg2‚0.00036874338295479726 trainer/global_step‚5849 % _timestamp‚1.7843453240116968e+09 _runtime‚4982.705300613 _step‚478 ' lr-AdamW/pg1‚0.00036874338295479726Þe›2ô- _wandbruntime‚4982  _runtime‚4982 Ñ'ç'õáO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162204 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚4982.706846081 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6245681809408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12572481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894590368367616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10249411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46340384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22821892 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6170795762084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46340386 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202150566575104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6245599523840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6180570932224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888409797435392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚4982.706846081 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6245681809408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12572481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894590368367616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10249411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46340384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22821892 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6170795762084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46340386 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202150566575104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6245599523840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6180570932224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888409797435392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4983.323001749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54171270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22825404 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚883990509936640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6171806815806 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12574251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6181583566336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195938395809792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6246605366784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10251153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894738622031360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202185001176576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242220032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚5849 % _timestamp‚1.7843453246292496e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6172147365808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161279 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6181583566336 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888557038465024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890162657302448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22825678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0CÎùñM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚883990509936640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6171806815806 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12574251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6181583566336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195938395809792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6246605366784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_bat#œ{gch_end/inactive_split.large_pool.freed‚10251153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894738622031360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202185001176576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242220032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚5849 % _timestamp‚1.7843453246292496e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6172147365808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161279 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6181583566336 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888557038465024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890162657302448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22825678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12574522 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202185118844416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4983.323001749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54171270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22825404 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23011955 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10336658 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203873212883968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5023.585565927 _step‚483 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756972544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6296789294080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203873212883968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5023.585565927 _step‚483 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756972544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6296789294080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23011955 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10336658 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23016011 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369014784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10338406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46729616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615731 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10338403 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23015746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23016011 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369014784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10338406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46729616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615731 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10338403 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23015746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5064.563505769 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6272028735900 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23202467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚487 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6347823314944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47103586 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281961935872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 % _timestamp‚1.7843454058667479e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55061131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12778288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205595769213952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205595666219008 MDDeviceStatsMonÝWÜOƒitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23202196 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903133883359232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12778556 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199247861516288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6281620596736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6347907697664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5064.563505769 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6272028735900 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23202467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚487 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6347823314944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47103586 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281961935872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 % _timestamp‚1.7843454058667479e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55061131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12778288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205595769213952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205595666219008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23202196 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903133883359232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12778556 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199247861516288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6281620596736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ç¯^2™- _wandbruntime‚5065  _runtime‚5065g\Fò&šìM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205630115369472 % _timestamp‚1.7843454064834647e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47111218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55080174 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903281124388864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 trainer/global_step‚5949 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6348843706880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23206308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904914957463464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874102 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23206038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070183 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544uü iI›ñM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070183 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205630115369472 % _timestamp‚1.7843454064834647e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47111218 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55080174 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903281124388864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 trainer/global_step‚5949 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6348843706880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23206308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904914957463464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874102 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23206038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224è3ǶÚ2œ- _wandbruntime‚5065  _runtime‚5065Ÿ»j trainer/global_step‚5949 1loss/SRqqt1/assignment_loss‚0.572284304727776 2loss/SRqqt2/assignment_loss‚0.4618407214899871 2loss/FBt2/assignment_loss‚0.010723948054154586 /loss/FBt2/detection_loss‚0.4955325473218839 0loss/FRt2/assignment_loss‚1.0002148690963946 /loss/FRt1/detection_loss‚0.6429276103571995 1loss/SRqqt2/detection_loss‚0.6706454996477303 /loss/FBt1/detection_loss‚0.4432947200892324 epoch‚14 _runtime‚5065.179645657 _step‚489 /loss/FRt1/assignment_loss‚1.600832787350468 /loss/FRt2/detection_loss‚1.2980471081079281 # _timestamp‚1.78434540648634e+09 3loss/FBt1/assignment_loss‚0.0011509659703625114 1loss/SRqqt1/detection_loss‚0.8722424149573863 %loss/total_loss‚8.069737497170504ÄœÕ ž† 3loss/FBt1/assignment_loss‚0.0011509659703625114 1loss/SRqqt1/detection_loss‚0.8722424149573863 %loss/total_loss‚8.069737497170504 trainer/global_step‚5949 1loss/SRqqt1/assignment_loss‚0.572284304727776 2loss/SRqqt2/assignment_loss‚0.4618407214899871 2loss/FBt2/assignment_loss‚0.010723948054154586 /loss/FBt2/detection_loss‚0.4955325473218839 0loss/FRt2/assignment_loss‚1.0002148690963946 /loss/FRt1/detection_loss‚0.6429276103571995 1loss/SRqqt2/detection_loss‚0.6706454996477303 /loss/FBt1/detection_loss‚0.4432947200892324 epoch‚14 _runtime‚5065.179645657 _step‚489 /loss/FRt1/assignment_loss‚1.600832787350468 /loss/FRt2/detection_loss‚1.2980471081079281 # _timestamp‚1.78434540648634e+09éƒ3œ2“Ÿ‚(:ˆ Ãæëҽ滔 gpu.0.gpu‚90.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÕãjÌ ‚(:Á ÃæëÒŽÓó•! disk.nvme0c0n1.in‚ 10262.34375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 494.265625( disk.nvme0c0n1.out‚16379.19775390625# disk.dm-0.out‚16379.19775390625& disk.nvme0n1.out‚16379.19775390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 10262.34375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.58203125 disk.drbd2165.in‚0) proc.memory.percent‚16.60630702972412 cpu‚26.11471419492362 disk.sda2.in‚0 disk.sda3.out‚ 163.88671875 network.sent‚ 9.357137e+06 network.recv‚ 1.386347e+06$ memory_percent‚96.98324203491211 disk.drbd2165.out‚0) disk./.usagePercent‚40.04612753992242$ disk./.usageGB‚596.7162399291992 disk.sda.out‚ 163.88671875 disk.nvme0n1.in‚ 10262.34375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 163.88671875# proc.memory.rssMB‚ 2720.77734375 disk.sda1.in‚0¢l-ñ“¡‚(:ˆ ÒæëÒÂÚ¯– gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'–Ô¢‚(:É ÒæëÒçݪ˜ disk.drbd2165.in‚0$ memory_percent‚96.98143005371094 network.sent‚ 9.420369e+06 network.recv‚ 1.391287e+06$ disk.nvme0c0n1.in‚10263.32421875 disk.sda.out‚ 165.51171875 disk.rbd1.in‚0 disk.rbd0.in‚0$ proc.memory.availableMB‚494.5625# disk.dm-0.out‚16419.62353515625 disk.sda2.in‚0 disk.rbd1.out‚ 12.64453125 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚596.8839149475098 disk.dm-0.in‚10263.32421875" disk.nvme0n1.in‚10263.32421875 disk.sda3.in‚0 disk.sda3.out‚ 165.51171875 disk.dm-1.out‚ 165.51171875 disk.sda2.out‚0) proc.memory.percent‚16.60630702972412( disk.nvme0c0n1.out‚16419.62353515625 disk.sda1.in‚0 cpu‚26.114645098110284) disk./.usagePercent‚40.05738035108996 disk.sda.in‚0& disk.nvme0n1.out‚16419.62353515625 disk.dm-1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2720.77734375 disk.sda1.out‚0fQ‡)‘£‚(:† áæëÒŸÍ·” gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ²]/Ò¤‚(:Ç áæëÒ”Òõ•# disk.dm-0.out‚16535.15185546875$ memory_percent‚96.98312282562256 network.sent‚ 9.425314e+06( proc.memory.availableMB‚ 494.28515625 disk.sda1.in‚0( disk.nvme0c0n1.out‚16535.15185546875 disk.nvme0n1.in‚ 10263.328125 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 10263.328125 disk.dm-0.in‚ 10263.328125# proc.memory.rssMB‚ 2720.77734375 network.recv‚ 1.394753e+06# disk./.usageGB‚596.896354675293 disk.sda3.out‚ 166.48046875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 166.48046875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚0& disk.nvme0n1.out‚16535.15185546875 cpu‚26.114384319146644* disk./.usagePercent‚40.058215191659116 disk.sda3.in‚0 disk.rbd1.out‚ 12.64453125 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 166.48046875) proc.memory.percent‚16.60630702972412 disk.drbd2165.out‚0OK502¥- _wandbruntime‚5105  _runtime‚5105\ò'+ƦÀ trainer/global_step‚5999 $ _timestamp‚1.784345446849824e+09 _runtime‚5105.543518638 _step‚490 & lr-AdamW/pg1‚0.0003645640548199449 & lr-AdamW/pg2‚0.0003645640548199449Î3…˧Š& lr-AdamW/pg1‚0.0003645640548199449 & lr-AdamW/pg2‚0.0003645640548199449 trainer/global_step‚5999 $ _timestamp‚1.784345446849824e+09 _runtime‚5105.543518638 _step‚490êÍ¿yN2¨- _wandbruntime‚5105  _runtime‚5105¼+ÿqé'©ãO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250939392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916828609317888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47485184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200919280830464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657437696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6322985772698 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47485186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10511158 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843454468508236e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55515583 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5105.544581965 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47485184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 trainer/global_step‚5999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚491 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207318291760640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6322645222808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200919299442688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515583 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200919280830464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657437696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6322985772698 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47485186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10511158 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843454468508236e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55515583 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5105.544581965 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47485184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 trainer/global_step‚5999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚491 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207318291760640ëÉÔ@22«- _wandbruntime‚5106  _runtime‚5106x—\õh¬ëM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524635 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910643192909824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚ 5106.15966389 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚5999 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543972 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47492818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47492818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12883873 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6400039632896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6333670071808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534626 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207352854306816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862981632 S?DeviceStatsMonitor.on_traijyð‰n_batch_end/requested_bytes.all.freed‚912290749954610 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10512906 % _timestamp‚1.7843454474658592e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6323996826532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916976862981632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6400126105600 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23396776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910643175870464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47492816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207352749221888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23396776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910643175870464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47492816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207352749221888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910643192909824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚ 5106.15966389 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚5999 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543972 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47492818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47492818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12883873 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6400039632896ìUMëí2®- _wandbruntime‚5106  _runtime‚5106AÞ#K ¯„ 0loss/FRt1/assignment_loss‚1.6959924505721373 2loss/SRqqt1/assignment_loss‚0.5164754530137938 3loss/FBt1/assignment_loss‚0.0008061140890272138 1loss/SRqqt1/detection_loss‚0.8536558606814582 /loss/FBt2/detection_loss‚0.5074633357195618 % _timestamp‚1.7843454474694085e+09 0loss/FRt2/assignment_loss‚0.8960478097600809 1loss/FBt2/assignment_loss‚0.00706087189467997 /loss/FRt1/detection_loss‚0.5906339785633912 2loss/SRqqt2/assignment_loss‚0.5009878167797023 /loss/FRt2/detection_loss‚1.3328001995380134 trainer/global_step‚5999 1loss/SRqqt2/detection_loss‚0.6279511828346862 /loss/FBt1/detection_loss‚0.7011924502750237 %loss/total_loss‚8.231067523721556 epoch‚14 _runtime‚5106.162720902 _step‚493IMö…°‰ 0loss/FRt2/assignment_loss‚0.8960478097600809 1loss/FBt2/assignment_loss‚0.00706087189467997 /loss/FRt1/detection_loss‚0.5906339785633912 2loss/SRqqt2/assignment_loss‚0.5009878167797023 /loss/FRt2/detection_loss‚1.3328001995380134 trainer/global_step‚5999 1loss/SRqqt2/detection_loss‚0.6279511828346862 /loss/FBt1/detection_loss‚0.7011924502750237 %loss/total_loss‚8.231067523721556 epoch‚14 _runtime‚5106.162720902 _step‚493 0loss/FRt1/assignment_loss‚1.6959924505721373 2loss/SRqqt1/assignment_loss‚0.5164754530137938 3loss/FBt1/assignment_loss‚0.0008061140890272138 1loss/SRqqt1/detection_loss‚0.8536558606814582 /loss/FBt2/detection_loss‚0.5074633357195618 % _timestamp‚1.7843454474694085e+09í<_S“±‚(:ˆ ðæëÒæÕ¾” gpu.0.gpu‚98.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù­äɲ‚(:¾ ðæëҙω– disk.sda3.out‚ 166.83984375 network.recv‚ 1.398327e+06 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 10329.90625) proc.memory.percent‚16.60630702972412$ disk./.usageGB‚597.0449409484863 disk.dm-1.out‚ 166.83984375 disk.rbd0.out‚0) disk./.usagePercent‚40.06818694112531# proc.memory.rssMB‚ 2720.77734375 network.sent‚ 9.430313e+06& disk.nvme0n1.out‚16599.89404296875 disk.sda1.in‚0$ memory_percent‚96.98333740234375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.70703125! disk.nvme0c0n1.in‚ 10329.90625 disk.dm-0.in‚ 10329.90625 disk.sda.out‚ 166.83984375 disk.sda3.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚16599.89404296875# disk.dm-0.out‚16599.89404296875 disk.sda2.in‚0 cpu‚26.118423238864175 disk.sda2.out‚0 disk.sda1.out‚0" proc.memory.availableMB‚494.25 disk.sda.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17õ*ù2³- _wandbruntime‚5124  _runtime‚5124Mé‹Êi5´—{ @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8345804810523987 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955342411994934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7509732246398926 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6486297845840454 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890133142471313 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 _runtime‚5124.769536294 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23040959239006042 jet/accuracy_2_of_5‚1 _step‚494 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07617727667093277 particle/accuracy_3_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39477765560150146 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6728231310844421 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7332760691642761 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8744234442710876 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7155329585075378 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20710285007953644 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27360081672668457 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6332083344459534 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958060383796692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407818913459778 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8747166395187378 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029967591166496277 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.664979875087738 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2033868134021759 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21115198731422424 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093742251396179 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04006079211831093 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5520833134651184 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 )jet/accuracy_2_of_3‚0.577369213104248 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8787301182746887 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5354695320129395 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 )jet/accuracy_1_of_2‚0.782569169998169 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22694873809814453 'particle/f_score‚0.6428744792938232 jet/accuracy_4_of_5‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.753231406211853 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_2_of_4‚0.809950590133667 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6832162737846375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949057698249817 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6567457914352417 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5110707879066467 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21586640179157257 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10128617286682129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_4‚0.9831644892692566 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.654087483882904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7577750086784363 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4963985085487366 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7091823816299438 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08360554277896881 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4765186309814453 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04630754515528679 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚8.039809246069822 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.668729305267334 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2204502820968628 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23077093064785004 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9476925730705261 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7121708989143372 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.816172182559967 (particle/accuracy‚0.8489776849746704 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888867139816284 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21719160676002502 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5602672696113586 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12022747099399567 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 $ _timestamp‚1.784345466075155e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34955358505249023 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933549761772156 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8325775861740112 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_3_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6885730028152466 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8744234442710876 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8155176043510437 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2836204767227173 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7035040259361267 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998043954372406 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026769254356622696 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7604570984840393 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8567649126052856 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03973744437098503 +jet/accuracy_3_of_4‚0.09240096062421799 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7302933931350708 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7003855109214783 *jet/accuracy_2_of_2‚0.1579238623380661 jet/accuracy_1_of_5‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.272289901971817 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.393542617559433 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6284655332565308 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4322666823863983 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027174469083547592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6621586084365845 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7576645612716675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8815843462944031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8680083751678467 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05097749084234238 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7541419267654419 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9134129881858826 +jet/accuracy_3_of_3‚0.07157667726278305 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22934865951538086 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48521551489830017 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5596317648887634 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10204677283763885 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6371586918830872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8575271368026733 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 jet/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *particle/sensitivity‚0.632987380027771 particle/accuracy_3_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/specificity‚0.9080485105514526 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7722090482711792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45139339566230774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020404089242219925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7014634609222412 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08814122527837753 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7808012366294861 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27407246828079224 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020404089242219925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22269171476364136 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7774701714515686 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30587172508239746 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2557435929775238 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 epoch‚14 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3289673924446106 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971703886985779 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.38689613342285156 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7626345753669739 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 jet/accuracy_3_of_5‚1 trainer/global_step‚5999 9"Purity/*FRt*SRqqt0FBt/evb0Bk4ent_purity‚0.3162715435028076 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7596394419670105 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.284329891204834 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (jet/accuracy_1_of_1‚0.35013347864151 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945900440216064 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26522713899612427 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0294138565659523 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.664979875087738 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21882005035877228 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2827877402305603 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27360081672668457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5640664100646973 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605574131011963 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1ÅÕz¢=µœ{ :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21882005035877228 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2827877402305603 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27360081672668457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5640664100646973 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605574131011963 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8345804810523987 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955342411994934 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7509732246398926 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6486297845840454 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890133142471313 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 _runtime‚5124.769536294 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23040959239006042 jet/accuracy_2_of_5‚1 _step‚494 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07617727667093277 particle/accuracy_3_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39477765560150146 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6728231310844421 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7332760691642761 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8744234442710876 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7155329585075378 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20710285007953644 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27360081672668457 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6332083344459534 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958060383796692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407818913459778 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8747166395187378 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029967591166496277 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.664979875087738 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2033868134021759 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21115198731422424 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093742251396179 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04006079211831093 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5520833134651184 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 )jet/accuracy_2_of_3‚0.577369213104248 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8787301182746887 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5354695320129395 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 )jet/accuracy_1_of_2‚0.782569169998169 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22694873809814453 'particle/f_score‚0.6428744792938232 jet/accuracy_4_of_5‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.753231406211853 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_2_of_4‚0.809950590133667 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6832162737846375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949057698249817 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6567457914352417 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5110707879066467 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21586640179157257 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10128617286682129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_4‚0.9831644892692566 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.654087483882904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7577750086784363 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4963985085487366 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7091823816299438 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08360554277896881 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4765186309814453 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04630754515528679 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚8.039809246069822 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.668729305267334 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2204502820968628 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23077093064785004 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9476925730705261 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7121708989143372 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.816172182559967 (particle/accuracy‚0.8489776849746704 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888867139816284 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21719160676002502 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5602672696113586 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12022747099399567 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 $ _timestamp‚1.784345466075155e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34955358505249023 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933549761772156 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8325775861740112 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_3_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6885730028152466 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8744234442710876 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8155176043510437 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2836204767227173 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7035040259361267 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998043954372406 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026769254356622696 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7604570984840393 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8567649126052856 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03973744437098503 +jet/accuracy_3_of_4‚0.09240096062421799 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7302933931350708 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7003855109214783 *jet/accuracy_2_of_2‚0.1579238623380661 jet/accuracy_1_of_5‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.272289901971817 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.393542617559433 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6284655332565308 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4322666823863983 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027174469083547592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6621586084365845 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7576645612716675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8815843462944031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8680083751678467 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05097749084234238 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7541419267654419 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9134129881858826 +jet/accuracy_3_of_3‚0.07157667726278305 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22934865951538086 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48521551489830017 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5596317648887634 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10204677283763885 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6371586918830872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8575271368026733 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 jet/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *particle/sensitivity‚0.632987380027771 particle/accuracy_3_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/specificity‚0.9080485105514526 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7722090482711792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45139339566230774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020404089242219925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7014634609222412 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08814122527837753 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7808012366294861 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27407246828079224 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020404089242219925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22269171476364136 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7774701714515686 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30587172508239746 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2557435929775238 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 epoch‚14 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3289673924446106 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971703886985779 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.38689613342285156 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7626345753669739 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 jet/accuracy_3_of_5‚1 trainer/global_step‚5999 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3162715435028076 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7596394419670105 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.284329891204834 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (jet/accuracy_1_of_1‚0.35013347864151 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945900440216064 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26522713899612427 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0294138565659523 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.664979875087738 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828îhc,„“¶‚(:ˆ ÿæëÒ±Ò›” gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚267.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®D5ÈÒ·‚(:Ç ÿæëÒ¦œ€— disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 741.7265625) disk./.usagePercent‚40.24540559662781$ disk.nvme0c0n1.in‚10338.98046875 disk.dm-0.in‚10339.44140625 disk.sda.in‚0 disk.sda3.out‚ 167.2734375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.70703125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.77734375 proc.cpu.threads‚17 disk.sda2.in‚0& disk.nvme0n1.out‚16860.82861328125$ memory_percent‚95.47286033630371 network.recv‚ 1.40378e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 167.2734375( disk.nvme0c0n1.out‚16860.82861328125 network.sent‚ 9.49335e+06# disk.dm-0.out‚16860.82861328125" disk.nvme0n1.in‚10339.23046875) proc.memory.percent‚16.60630702972412$ disk./.usageGB‚599.6856269836426 disk.drbd2165.out‚0 disk.sda.out‚ 167.2734375 cpu‚26.121994724133852漢m“¸‚(:ˆ ŽçëÒÛ­” gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.633' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pã–Õ¹‚(:Ê ŽçëÒœ¡•) proc.memory.percent‚16.60630702972412 network.recv‚ 1.4087e+06 disk.sda3.out‚ 167.85546875# proc.memory.rssMB‚ 2720.77734375( disk.nvme0c0n1.out‚17097.78662109375& disk.nvme0n1.out‚17097.78662109375 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚10504.68359375 disk.dm-1.out‚ 167.85546875 disk.drbd2165.in‚0 cpu‚26.121610862393403 network.sent‚ 9.550652e+06 disk.drbd2165.out‚0$ disk./.usageGB‚599.9104347229004 disk.sda.in‚0$ memory_percent‚95.47295570373535# disk.dm-0.out‚17097.78662109375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.76953125 disk.sda1.out‚0' proc.memory.availableMB‚ 741.7109375) disk./.usagePercent‚40.26049263263566 disk.dm-0.in‚10504.68359375 disk.sda.out‚ 167.85546875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚10504.68359375 proc.cpu.threads‚17 disk.rbd0.out‚0Ðxî_“º‚(:ˆ çëҨ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O]­õÆ»‚(:» çëÒÔÆ • disk.dm-0.out‚18623.03515625 disk.sda2.out‚0 cpu‚26.121018665165114 network.sent‚ 9.555786e+06 disk.sda1.in‚0 disk.dm-1.out‚ 168.28125% disk.nvme0c0n1.out‚18623.03515625( proc.memory.availableMB‚ 741.72265625# disk.nvme0n1.out‚18623.03515625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.60630702972412 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚10505.52734375 disk.rbd1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚10505.52734375# proc.memory.rssMB‚ 2720.77734375 disk.rbd1.out‚ 12.76953125 disk.drbd2165.out‚0 network.recv‚ 1.412201e+06# disk./.usageGB‚599.836067199707 disk.sda3.out‚ 168.28125$ memory_percent‚95.47288417816162 disk.sda.out‚ 168.28125* disk./.usagePercent‚40.255501765755646 disk.rbd0.out‚0 disk.dm-0.in‚10505.52734375ß`’2¼- _wandbruntime‚5167  _runtime‚5167Ex1§Å½¿ % lr-AdamW/pg1‚0.000363152838390128 % lr-AdamW/pg2‚0.000363152838390128 trainer/global_step‚6049 % _timestamp‚1.7843455084497046e+09 _runtime‚5167.143293432 _step‚495ˆÇážÊ¾Ä _runtime‚5167.143293432 _step‚495 % lr-AdamW/pg1‚0.000363152838390128 % lr-AdamW/pg2‚0.000363152838390128 trainer/global_step‚6049 % _timestamp‚1.7843455084497046e+09ï« H2¿- _wandbruntime‚5167  _runtime‚5167>äÕ¶'ÀâO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56040362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209075047772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚915054796853248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23618892 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10600928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6468501298176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6391321984840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47987450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919787552972800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202606649468928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10600931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6400744808448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6390981434950 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926188656159744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13017694 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47987452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202606630856704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚915054779813888 % _timestamp‚1.7843455084505587e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401086147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6400744808448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209075150767104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104037791 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919787552972800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23618622 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batcb)ª2h_start/reserved_bytes.small_pool.peak‚ 564133888@H8Îí'ÁçO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10600928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6468501298176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6391321984840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47987450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919787552972800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202606649468928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10600931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6400744808448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6390981434950 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926188656159744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13017694 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47987452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202606630856704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚915054779813888 % _timestamp‚1.7843455084505587e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401086147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6400744808448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209075150767104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104037791 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919787552972800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23618622 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56040362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209075047772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚915054796853248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23618892ðØúšK2Â- _wandbruntime‚5167  _runtime‚5167xÚò&ÃìM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6469525492224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚497 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47995082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚926336909823488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202640059615232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47995082 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202640078227456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23622430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921593282974432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚915201307525120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56049414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6402098781696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47995084 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚5167.760529301 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919934811041792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚915201290485760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚919934794002432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209109603719680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚919934794002432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚926336551437824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104044496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10602676 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47995084 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10602673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6401757435392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23622696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23622430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921593282974432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚915201307525120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56049414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6402098781696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47995084 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚5167.760529301 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919934811041792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚915201290485760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚919934794002432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209109603719680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚919934794002432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚926336551437824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104044496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10602676 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47995084 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10602673 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6401757435392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23622696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚497 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47995082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚926336909823488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202640059615232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ñ#;ðQ2Å- _wandbruntime‚5167  _runtime‚5167£µb; Æ„ 3loss/FBt1/assignment_loss‚0.0010695508306877155 1loss/SRqqt1/detection_loss‚1.0323926560442023 trainer/global_step‚6049 _step‚498 epoch‚15 2loss/SRqqt1/assignment_loss‚0.5574141978048835 2loss/SRqqt2/assignment_loss‚0.5036324674158956 1loss/FBt2/assignment_loss‚0.03190005685921878 /loss/FBt1/detection_loss‚0.5239030451339204 /loss/FBt2/detection_loss‚0.4969649959145294 % _timestamp‚1.7843455090699074e+09 _runtime‚5167.763265917 0loss/FRt1/assignment_loss‚1.7189060704959067 /loss/FRt1/detection_loss‚0.5761261217953448 /loss/FRt2/detection_loss‚1.2565219533633714 1loss/SRqqt2/detection_loss‚0.6123386045926127 %loss/total_loss‚8.128246911299481 0loss/FRt2/assignment_loss‚0.8170771910489061ô?àlj epoch‚15 2loss/SRqqt1/assignment_loss‚0.5574141978048835 2loss/SRqqt2/assignment_loss‚0.5036324674158956 1loss/FBt2/assignment_loss‚0.03190005685921878 /loss/FBt1/detection_loss‚0.5239030451339204 /loss/FBt2/detection_loss‚0.4969649959145294 % _timestamp‚1.7843455090699074e+09 _runtime‚5167.763265917 0loss/FRt1/assignment_loss‚1.7189060704959067 /loss/FRt1/detection_loss‚0.5761261217953448 /loss/FRt2/detection_loss‚1.2565219533633714 1loss/SRqqt2/detection_loss‚0.6123386045926127 %loss/total_loss‚8.128246911299481 0loss/FRt2/assignment_loss‚0.8170771910489061 3loss/FBt1/assignment_loss‚0.0010695508306877155 1loss/SRqqt1/detection_loss‚1.0323926560442023 trainer/global_step‚6049 _step‚498òcÇÁœÈ‚(:‘ ¬çëҢ뫓 gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.46' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.69199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆG€0™É‚(:¿ ¬çëÒô¬£— disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚19680.41943359375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚19680.41943359375 disk.rbd1.out‚ 12.89453125# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 disk.rbd0.in‚0 disk.dm-1.out‚ 168.78125# disk.nvme0c0n1.in‚ 10506.4921875 disk.rbd1.in‚0# memory_percent‚95ÜŽM±1.4730749130249$ disk./.usageGB‚599.7108688354492 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 1.415442e+06) disk./.usagePercent‚40.24709959848188 disk.drbd2165.in‚0 disk.dm-0.in‚ 10506.4921875 disk.sda.in‚0( proc.memory.availableMB‚ 741.69140625 disk.drbd2165.out‚0# disk.dm-0.out‚19680.41943359375 cpu‚26.12074478835087 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.out‚ 168.78125! disk.nvme0n1.in‚ 10506.4921875 disk.sda3.in‚0 disk.sda3.out‚ 168.78125 network.sent‚ 9.560772e+06P¬%“Ê‚(:ˆ »çëÒþÃÿ“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚267.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÙ’TÓË‚(:È »çëÒµÒ—$ disk./.usageGB‚599.5922546386719& disk.nvme0n1.out‚19923.86962890625 disk.rbd0.out‚0 network.recv‚ 1.420771e+06# disk.dm-0.out‚19923.86962890625 disk.sda.in‚0 disk.sda3.out‚ 169.16796875 cpu‚26.1204481780768 disk.sda1.in‚0 disk.dm-1.out‚ 169.16796875 network.sent‚ 9.62372e+06* disk./.usagePercent‚40.239139300212216 disk.sda.out‚ 169.16796875! disk.nvme0n1.in‚ 10506.4921875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 10506.4921875 disk.rbd1.out‚ 12.89453125# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412( disk.nvme0c0n1.out‚19923.86962890625 disk.dm-0.in‚ 10506.4921875 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.in‚0$ memory_percent‚95.47312259674072( proc.memory.availableMB‚ 741.68359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0œ–Á’Ì‚(:‡ ÊçëÒ®Ž¬“ gpu.0.gpu‚20.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚281.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<WŽÉÍ‚(:¾ ÊçëÒ‚— disk.sda.in‚0 disk.sda3.out‚ 169.48046875$ memory_percent‚95.47290802001953% proc.memory.availableMB‚ 741.71875) proc.memory.percent‚16.60630702972412$ disk./.usageGB‚599.5920715332031" disk.nvme0n1.out‚ 19987.2421875) disk./.usagePercent‚40.23912701185057 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚10677.28515625 disk.dm-0.in‚10677.28515625 disk.sda.out‚ 169.48046875 cpu‚26.120149609569044 network.sent‚ 9.628664e+06 disk.dm-1.out‚ 169.48046875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 19987.2421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 19987.2421875 disk.sda3.in‚0 disk.rbd1.out‚ 12.95703125 proc.cpu.threads‚17" disk.nvme0n1.in‚10677.28515625 network.recv‚ 1.424011e+06×!2Î- _wandbruntime‚5207  _runtime‚5207¶Ð¾ÆÏÀ & lr-AdamW/pg1‚0.0003617327190185367 & lr-AdamW/pg2‚0.0003617327190185367 trainer/global_step‚6099 $ _timestamp‚1.784345549211663e+09 _runtime‚5207.908409863 _step‚499‹ûø±ËÐÅ trainer/global_step‚6099 $ _timestamp‚1.784345549211663e+09 _runtime‚5207.908409863 _step‚499 & lr-AdamW/pg1‚0.0003617327190185367 & lr-AdamW/pg2‚0.0003617327190185367óˆàå2Ñ- _wandbruntime‚5207  _runtime‚5207µa£Éç'ÒáO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56504791 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210797759600640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚500 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933601044764672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10688181 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210797864692736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204278087395328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6441938471748 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚927149604454400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚927149621493760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚927149621493760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6451781649408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚928822268918596 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13122184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6441597921858 trainer/global_step‚6099 % _timestamp‚1.7843455492156792e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23810362 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933601044764672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933601403143168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚922380313407488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48369052 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6519690817536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48369050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56494814 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48369052 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56504791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104863864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6451440310272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204278068783104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23810622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6451440310272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13122441 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48369050 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚922380330446848 _runtime‚ 5207.90931911 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933601403143168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56494814 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6451781649408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169tøFì'ÓæO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933601044764672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10688181 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210797864692736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204278087395328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6441938471748 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚927149604454400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚927149621493760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚927149621493760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6451781649408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚928822268918596 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13122184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6441597921858 trainer/global_step‚6099 % _timestamp‚1.7843455492156792e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23810362 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933601044764672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933601403143168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚922380313407488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48369052 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6519690817536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48369050 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56494814 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48369052 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56504791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104863864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6451440310272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204278068783104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23810622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6451440310272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13122441 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48369050 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚922380330446848 _runtime‚ 5207.90931911 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933601403143168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56494814 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6451781649408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56504791 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210797759600640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚500ô¤n12Ô- _wandbruntime‚5208  _runtime‚5208PGŒ€¹ ÕìM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6452452937216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚927296862523392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933749656806912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚922526824079360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933749656806912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚927296845484032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚927296862523392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚501 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48376682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6452794283520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210832314499584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10689923 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6520798345728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48376684 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48376684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚922526841118720 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6442608975580 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚6099 % _timestamp‚1.7843455500189965e+09 _runtime‚5208.712777068 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚927296845484032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104880548 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13124246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6452452937216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6442949525582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104890541 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56503866 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204311516153856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10689926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23814452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23814169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚928969790644302 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210832203123200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6520705581568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56513857 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚928969433054940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513857 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664’*÷&ÖñM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚927296845484032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚927296862523392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚501 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48376682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6452794283520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210832314499584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10689923 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6520798345728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48376684 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48376684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚922526841118720 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6442608975580 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚6099 % _timestamp‚1.7843455500189965e+09 _runtime‚5208.712777068 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚927296845484032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104880548 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13124246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6452452937216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6442949525582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104890541 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56503866 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204311516153856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10689926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23814452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23814169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚928969790644302 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210832203123200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6520705581568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56513857 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚928969433054940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513857 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6452452937216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚927296862523392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933749656806912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚922526824079360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933749656806912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088õÐc÷{2×- _wandbruntime‚5208  _runtime‚5208‡k3úØ‚ /loss/FRt1/detection_loss‚0.5802325417324928 /loss/FRt2/detection_loss‚1.3376359287691226 epoch‚15 _runtime‚5208.715543309 /loss/FRt2/assignment_loss‚0.893926819002359 1loss/SRqqt2/assignment_loss‚0.528690488272689 /loss/FBt1/detection_loss‚0.9680624871418394 % _timestamp‚1.7843455500219522e+09 _step‚502 3loss/FBt1/assignment_loss‚0.0029152389083594798 0loss/SRqqt1/detection_loss‚0.937050622396378 1loss/SRqqt2/detection_loss‚0.6673553897495539 %loss/total_loss‚8.860902900500857 trainer/global_step‚6099 0loss/FRt1/assignment_loss‚1.6986571940985116 2loss/SRqqt1/assignment_loss‚0.6698246780572553 2loss/FBt2/assignment_loss‚0.012828323881323597 /loss/FBt2/detection_loss‚0.5637231884909732xÜ¬Þ Ù‡ 1loss/SRqqt2/detection_loss‚0.6673553897495539 %loss/total_loss‚8.860902900500857 trainer/global_step‚6099 0loss/FRt1/assignment_loss‚1.6986571940985116 2loss/SRqqt1/assignment_loss‚0.6698246780572553 2loss/FBt2/assignment_loss‚0.012828323881323597 /loss/FBt2/detection_loss‚0.5637231884909732 /loss/FRt1/detection_loss‚0.5802325417324928 /loss/FRt2/detection_loss‚1.3376359287691226 epoch‚15 _runtime‚5208.715543309 /loss/FRt2/assignment_loss‚0.893926819002359 1loss/SRqqt2/assignment_loss‚0.528690488272689 /loss/FBt1/detection_loss‚0.9680624871418394 % _timestamp‚1.7843455500219522e+09 _step‚502 3loss/FBt1/assignment_loss‚0.0029152389083594798 0loss/SRqqt1/detection_loss‚0.937050622396378öuéWðÚ‚(:’ ÙçëÒÒ— ” gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.209' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.84179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«"RáÈÛ‚(:½ ÙçëÒã­È— network.sent‚ 9.64658e+06 disk.dm-1.out‚ 169.6484375 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚95.47500610351562 disk.sda3.in‚0 disk.sda3.out‚ 169.6484375 disk.rbd0.out‚0 disk.sda.out‚ 169.6484375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚10677.28515625# proc.memory.availableMB‚741.375) disk./.usagePercent‚40.23918512556084" disk.nvme0n1.in‚10677.28515625 disk.rbd1.in‚0 disk.rbd1.out‚ 12.95703125 disk.sda1.out‚0! disk.dm-0.out‚19989.955078125 network.recv‚ 1.427564e+06$ disk.nvme0n1.out‚19989.955078125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.60630702972412 cpu‚26.11989679552517$ disk.nvme0c0n1.in‚10677.28515625# proc.memory.rssMB‚ 2720.77734375$ disk./.usageGB‚599.5929374694824& disk.nvme0c0n1.out‚19989.955078125 disk.sda.in‚0 disk.sda2.out‚0ø4™#“Ü‚(:ˆ èçëÒíü“ gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ䵼݂(:± èçëÒáØõ• disk.dm-0.in‚10678.25 disk.sda2.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0c0n1.in‚10678.25( disk.nvme0c0n1.out‚19998.19287109375# disk.dm-0.out‚19998.19287109375 disk.sda1.in‚0 disk.dm-1.out‚ 169.84375 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 13.01953125 disk.dm-1.in‚0& disk.nvme0n1.out‚19998.19287109375 disk.sda3.out‚ 169.84375 cpu‚26.119429702486098 network.recv‚ 1.432497e+06) disk./.usagePercent‚40.23918154145536 disk.sda1.out‚0) proc.memory.percent‚16.60630702972412$ disk./.usageGB‚599.5928840637207 disk.rbd0.out‚0$ memory_percent‚95.47314643859863 disk.sda.in‚0 disk.sda.out‚ 169.84375 disk.sda2.in‚0 disk.nvme0n1.in‚10678.25' proc.memory.availableMB‚ 741.6796875# proc.memory.rssMB‚ 2720.77734375 network.sent‚ 9.696482e+06 disk.drbd2165.out‚0bàÝ#2Þ- _wandbruntime‚5248  _runtime‚5248Ö[CûÇßÁ & lr-AdamW/pg1‚0.0003603037879167847 & lr-AdamW/pg2‚0.0003603037879167847 trainer/global_step‚6149 % _timestamp‚1.7843455901877694e+09 _runtime‚5248.881330209 _step‚503C5‚ÎÌàÆ _runtime‚5248.881330209 _step‚503 & lr-AdamW/pg1‚0.0003603037879167847 & lr-AdamW/pg2‚0.0003603037879167847 trainer/global_step‚6149 % _timestamp‚1.7843455901877694e+09÷A.áê2á- _wandbruntime‚5248  _runtime‚5248¯±ß¤è'ââO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941013791748096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5248.882571182 _step‚504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941013791748096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56949266 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56959243 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934511655936000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6570837050880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚6149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6502477151232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10775431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10775428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205949506709504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6570746376704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936198061409854 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941014150126592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476š¢gãçO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205949506709504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6570746376704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936198061409854 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941014150126592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_reje™Q`ctions‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941013791748096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5248.882571182 _step‚504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941013791748096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56949266 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56959243 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934511655936000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6570837050880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚6149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6502477151232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10775431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10775428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ø ¼*'2ä- _wandbruntime‚5249  _runtime‚5249¥°`ò&åìM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934658914004992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚936345940724810 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6503148439040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6493225462488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5249.498322918 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48758282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10777173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212554807923712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56958318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56958318 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13229550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6571853843456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56968309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843455908044887e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105726593 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24006726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941162403790336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24006461 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚505 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212554694450176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚505 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212554694450176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56958318 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13229550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6571853843456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56968309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843455908044887e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105726593 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24006726 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941162403790336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24006461 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F”Øt2ð- _wandbruntime‚5289  _runtime‚5289Ò3ÝÉñà ' lr-AdamW/pg1‚0.00035886613686244645 ' lr-AdamW/pg2‚0.00035886613686244645 trainer/global_step‚6199 % _timestamp‚1.7843456312257793e+09 _runtime‚5289.919395917 _step‚507q’éÎòÈ ' lr-AdamW/pg1‚0.00035886613686244645 ' lr-AdamW/pg2‚0.00035886613686244645 trainer/global_step‚6199 % _timestamp‚1.7843456312257793e+09 _runtime‚5289.919395917 _step‚507ûJãA>2ó- _wandbruntime‚5289  _runtime‚5289ð!j¡|ôâO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6552831313920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948426538731520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948426897110016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10862681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10862678 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚943574569079612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.o¢—1­ln_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24194902 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106545947 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6543171445564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13332494 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6553172653056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24195175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚6199 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6621863545856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57413695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚941873724456960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13332224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207620944635904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚941873724456960 _runtime‚5289.920713184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49132250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937031397634048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937031380594688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214242826793984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6542830895674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6621768677376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚941873707417600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843456312269485e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49132250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49132252 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6552831313920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚941873707417600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948426897110016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6553172653056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57413695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2~Á!í'õçO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49132250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49132252 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6552831313920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚941873707417600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948426897110016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6553172653056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57413695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6552831313920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948426538731520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948426897110016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10862681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10862678 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚943574569079612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24194902 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106545947 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6543171445564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13332494 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6553172653056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24195175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚6199 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6621863545856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57413695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚941873724456960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13332224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207620944635904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚941873724456960 _runtime‚5289.920713184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49132250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937031397634048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937031380594688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214242826793984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6542830895674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6621768677376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚941873707417600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843456312269485e+09üYKæ2ö- _wandbruntime‚5290  _runtime‚5290£HÛð&÷êM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13334601 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942020965486592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207654373394432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948574792388096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57422761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214277272791040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚509 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106552652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942020965486592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6553843940864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6622880784384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49139884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937177891266560 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198748 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6554185287168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6553843940864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49139882 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10864426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13334325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214277155123200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6544182499398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942020948447232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937177908305920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6554185287168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106552652 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57412770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚5290.536448445 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6622781728768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚943722090805318 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49139884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚6199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10864423 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚943721733215956 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24199027 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207654392006656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6554185287168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106552652 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57412770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚5290.536448445 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6622781728768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚943722090805318 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49139884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49139882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 trainer/global_step‚6199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10864423 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚943721733215956 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24199027 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207654392006656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942020965486592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207654373394432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948574792388096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57422761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214277272791040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚509 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106552652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942020965486592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6553843940864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6622880784384 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49139884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937177891266560 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198748 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6554185287168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6553843940864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49139882 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10864426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13334325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214277155123200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6544182499398 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942020948447232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937177908305920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949235758899200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944356914188288 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944356931227648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10949928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950950719160120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49513852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209292401174528 trainer/global_step‚6249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955839285714944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6672949262336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209292382562304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6672860685312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49513850 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24387080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215965350436864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49513850 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955839644093440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955839644093440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6603526815744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107381999 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6593447382582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚5330.326564752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57858170 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949235758899200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949235758899200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944356914188288 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944356931227648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10949928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950950719160120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49513852 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209292401174528 trainer/global_step‚6249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955839285714944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6672949262336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209292382562304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6672860685312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49513850 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24387080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215965350436864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49513850 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955839644093440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955839644093440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6603526815744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107381999 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6593447382582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚5330.326564752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57858170 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949235758899200 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6594458436304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49521484 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24390945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13439001 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6604539442688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6673886832128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57877213 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107388704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6594798986306 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6604880788992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚944503441899520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10951673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49521484 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57877213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215999698152960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209325811320832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49521482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949382999928832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6604880788992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215999801140736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955987897757184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949383016968192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13439269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚5330.940604134 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚944503424860160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24390674 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚513 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57867222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843456722468433e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚949382999928832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10951676 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57867222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955987539371520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951098240885826|û¬÷&ŠñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚949383016968192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209325829933056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951097883296464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6594458436304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49521484 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24390945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13439001 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6604539442688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6673886832128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57877213 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107388704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6594798986306 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6604880788992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚944503441899520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10951673 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49521484 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57877213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215999698152960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209325811320832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49521482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949382999928832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6604880788992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215999801140736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955987897757184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949383016968192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13439269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚5330.940604134 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚944503424860160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24390674 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚513 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57867222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843456722468433e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚949382999928832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10951676 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57867222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955987539371520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚9510982408858268«;Ž2‹- _wandbruntime‚5330  _runtime‚5330\׆Œ‚ 2loss/SRqqt1/assignment_loss‚0.6060049664479448 /loss/FRt2/detection_loss‚1.2287984740687534 1loss/SRqqt1/detection_loss‚0.8409954447206109 /loss/FBt1/detection_loss‚0.7735751030906555 %loss/total_loss‚8.241062198565304 0loss/FRt2/assignment_loss‚0.9429065277159565 trainer/global_step‚6249 .loss/FRt1/detection_loss‚0.557650055805703 .loss/FBt2/detection_loss‚0.386777853152643 % _timestamp‚1.7843456722500455e+09 _runtime‚5330.943363954 _step‚514 0loss/FRt1/assignment_loss‚1.7705204475910292 2loss/SRqqt2/assignment_loss‚0.4882453993316199 4loss/FBt1/assignment_loss‚0.00023679268207777956 1loss/FBt2/assignment_loss‚0.00352147900955362 0loss/SRqqt2/detection_loss‚0.641829654948756 epoch‚15æ=K. ‡ .loss/FRt1/detection_loss‚0.557650055805703 .loss/FBt2/detection_loss‚0.386777853152643 % _timestamp‚1.7843456722500455e+09 _runtime‚5330.943363954 _step‚514 0loss/FRt1/assignment_loss‚1.7705204475910292 2loss/SRqqt2/assignment_loss‚0.4882453993316199 4loss/FBt1/assignment_loss‚0.00023679268207777956 1loss/FBt2/assignment_loss‚0.00352147900955362 0loss/SRqqt2/detection_loss‚0.641829654948756 epoch‚15 2loss/SRqqt1/assignment_loss‚0.6060049664479448 /loss/FRt2/detection_loss‚1.2287984740687534 1loss/SRqqt1/detection_loss‚0.8409954447206109 /loss/FBt1/detection_loss‚0.7735751030906555 %loss/total_loss‚8.241062198565304 0loss/FRt2/assignment_loss‚0.9429065277159565 trainer/global_step‚6249‚ÉÕ‘‰žŽ‚(:“ ÑèëÒ–ùù“ gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.931' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.786199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´©öÖ‚(:Ë ÑèëÒ“¯ƒ– network.recv‚ 1.459997e+06 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚26.121376909665553) disk./.usagePercent‚40.25344448921049 disk.rbd1.out‚ 13.20703125 disk.sda1.in‚0& proc.memory.availableMB‚ 495.046875 disk.sda.out‚ 174.54296875& disk.nvme0n1.out‚21159.77001953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚21159.77001953125 disk.dm-0.in‚10942.51171875# proc.memory.rssMB‚ 2720.77734375 proc.cpu.threads‚17" disk.nvme0n1.in‚10942.51171875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 174.54296875 disk.rbd1.in‚0 disk.dm-1.out‚ 174.54296875 disk.drbd2165.out‚0# disk.dm-0.out‚21159.77001953125) proc.memory.percent‚16.60630702972412$ memory_percent‚96.97847366333008 network.sent‚ 9.847467e+06$ disk./.usageGB‚599.8054122924805$ disk.nvme0c0n1.in‚10942.51171875 disk.sda.in‚0“o“‚(:ˆ àèëÒØåÄ” gpu.0.gpu‚88.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚241.571' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3142 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÍ%Ñ‘‚(:Æ àèëÒœ¢— network.sent‚ 9.910315e+06 disk.sda1.in‚0$ disk.nvme0c0n1.in‚10942.51171875$ memory_percent‚96.97542190551758 network.recv‚ 1.465001e+06 disk.sda.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0) disk./.usagePercent‚40.25342631267556& disk.nvme0n1.out‚21166.92822265625 disk.dm-1.in‚0 cpu‚26.1220942110024 proc.cpu.threads‚17$ disk./.usageGB‚599.8051414489746( disk.nvme0c0n1.out‚21166.92822265625& proc.memory.availableMB‚ 495.546875" disk.nvme0n1.in‚10942.51171875 disk.sda3.out‚ 174.7578125 disk.rbd0.in‚0 disk.dm-1.out‚ 174.7578125 disk.dm-0.in‚10942.51171875 disk.sda.out‚ 174.7578125 disk.rbd1.out‚ 13.26953125 disk.drbd2165.in‚0) proc.memory.percent‚16.60630702972412 disk.sda2.in‚0# disk.dm-0.out‚21166.92822265625# proc.memory.rssMB‚ 2720.77734375¡ÞS“’‚(:ˆ ïèëÒ°Óû” gpu.0.gpu‚78.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›O7Õ“‚(:Ê ïèëÒø„Í• network.sent‚ 9.915384e+06 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚21175.56787109375 disk.sda2.out‚0 disk.sda3.out‚ 175.16015625 disk.dm-1.out‚ 175.16015625( disk.nvme0c0n1.out‚21175.56787109375( proc.memory.availableMB‚ 495.79296875! disk.nvme0n1.in‚ 10943.4765625) proc.memory.percent‚16.60630702972412 cpu‚26.12274426213579 network.recv‚ 1.468535e+06* disk./.usagePercent‚40.253429640773504 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 13.26953125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 10943.4765625 disk.sda2.in‚0$ memory_percent‚96.97391986846924 disk.drbd2165.out‚0$ disk./.usageGB‚599.8051910400391 disk.sda1.in‚0 disk.sda.out‚ 175.16015625& disk.nvme0n1.out‚21175.56787109375 disk.dm-0.in‚ 10943.4765625 disk.sda.in‚0œÛ2”- _wandbruntime‚5370  _runtime‚5370_$t9É•Ã _step‚515 ' lr-AdamW/pg1‚0.00035596504480070977 ' lr-AdamW/pg2‚0.00035596504480070977 trainer/global_step‚6299 % _timestamp‚1.7843457121957426e+09 _runtime‚5370.889385096£o¯õÎ–È _step‚515 ' lr-AdamW/pg1‚0.00035596504480070977 ' lr-AdamW/pg2‚0.00035596504480070977 trainer/global_step‚6299 % _timestamp‚1.7843457121957426e+09 _runtime‚5370.889385096ƒkm‘2—- _wandbruntime‚5370  _runtime‚5370Ð6%“˜áO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58312622 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217687818091008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958326511651378 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13540801 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚956597827420160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49895452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963252032698368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24577979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843457121968532e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958326869240628 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6654563656704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654222317568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6644404419380 XJD@#ðJ eviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963252032698368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24577979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843457121968532e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958326869240628 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6654563656704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654222317568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6644404419380 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚956745068449792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6724913942528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108224756 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49903084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚956745068449792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚951828975493120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11038926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11038923 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚963400644740608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6655576290816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217722163189760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655234944512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24581761 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49903084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58321674 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6655576290816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956745051410432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24582018 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958474033376972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 5371.50489041 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 238425210882Æ°/„#œðM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11038926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11038923 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚963400644740608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6655576290816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217722163189760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655234944512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24581761 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49903084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58321674 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6655576290816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956745051410432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24582018 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958474033376972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 5371.50489041 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚963400286354944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210997249247232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49903082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚956745068449792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6724913942528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108224756 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49903084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚956745068449792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚951828975493120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0…ÂSCç2- _wandbruntime‚5371  _runtime‚5371pi’Æ žƒ /loss/FRt1/assignment_loss‚1.668879846849385 .loss/FRt1/detection_loss‚0.562348815122889 epoch‚15 _runtime‚5371.507218175 1loss/SRqqt1/assignment_loss‚0.571330083635305 2loss/SRqqt2/assignment_loss‚0.4793389646119613 3loss/FBt1/assignment_loss‚0.0001593014499295597 3loss/FBt2/assignment_loss‚0.0008248248243593646 /loss/FBt2/detection_loss‚0.5900405491075407 0loss/FRt2/assignment_loss‚0.8760447336085869 /loss/FRt2/detection_loss‚1.2828766032685517 1loss/SRqqt1/detection_loss‚0.9170864759028324 /loss/FBt1/detection_loss‚0.7968903003774327 %loss/total_loss‚8.375447709676056 1loss/SRqqt2/detection_loss‚0.6296272109172839 trainer/global_step‚6299 % _timestamp‚1.7843457128138983e+09 _step‚518aÖàlŸˆ % _timestamp‚1.7843457128138983e+09 _step‚518 /loss/FRt1/assignment_loss‚1.668879846849385 .loss/FRt1/detection_loss‚0.562348815122889 epoch‚15 _runtime‚5371.507218175 1loss/SRqqt1/assignment_loss‚0.571330083635305 2loss/SRqqt2/assignment_loss‚0.4793389646119613 3loss/FBt1/assignment_loss‚0.0001593014499295597 3loss/FBt2/assignment_loss‚0.0008248248243593646 /loss/FBt2/detection_loss‚0.5900405491075407 0loss/FRt2/assignment_loss‚0.8760447336085869 /loss/FRt2/detection_loss‚1.2828766032685517 1loss/SRqqt1/detection_loss‚0.9170864759028324 /loss/FBt1/detection_loss‚0.7968903003774327 %loss/total_loss‚8.375447709676056 1loss/SRqqt2/detection_loss‚0.6296272109172839 trainer/global_step‚6299†î[až ‚(:“ þèëÒŒŠ×” gpu.0.gpu‚87.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚228.449' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.689800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X#ýÖ¡‚(:Ë þèëÒÑÑ — disk.dm-0.in‚11067.69140625# proc.memory.rssMB‚ 2720.77734375$ disk./.usageGB‚599.8053817749023 disk.sda.out‚ 175.44140625& disk.nvme0n1.out‚21186.32568359375 disk.sda3.out‚ 175.44140625 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 network.recv‚ 1.47183e+06 disk.sda1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚21186.32568359375) proc.memory.percent‚16.60630702972412 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚11067.69140625 disk.rbd1.out‚ 13.33203125 disk.dm-1.out‚ 175.44140625 disk.drbd2165.in‚0# memory_percent‚96.9700574874878 proc.cpu.threads‚17) disk./.usagePercent‚40.25344244115022( proc.memory.availableMB‚ 496.42578125 disk.rbd1.in‚0# disk.dm-0.out‚21186.32568359375 cpu‚26.123357337028338 network.sent‚ 9.933305e+06 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚11067.69140625=7)C’¢‚(:‡ éëÒ߇ç“ gpu.0.gpu‚77.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.113 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"b (Σ‚(:à éëÒâ¦Ð•( disk.nvme0c0n1.out‚21190.02978515625 network.recv‚ 1.476227e+06 disk.rbd0.in‚0( proc.memory.availableMB‚ 496.67578125# proc.memory.rssMB‚ 2720.77734375 network.sent‚ 9.982208e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 175.78125 cpu‚26.12390774406415 disk.sda3.in‚0 disk.rbd1.out‚ 13.33203125 disk.drbd2165.in‚0 disk.sda.out‚ 175.78125 disk.drbd2165.out‚0 disk.dm-0.in‚11067.69140625) disk./.usagePercent‚40.25344244115022" disk.nvme0n1.in‚11067.69140625 disk.sda3.out‚ 175.78125 disk.rbd1.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚21190.02978515625 disk.sda.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚21190.02978515625 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.96853160858154$ disk./.usageGB‚599.8053817749023$ disk.nvme0c0n1.in‚11067.69140625 disk.sda2.in‚0) proc.memory.percent‚16.60630702972412Œù¼{2¤- _wandbruntime‚5411  _runtime‚5411‹‡ÐÆ¥À % _timestamp‚1.7843457527710369e+09 _runtime‚ 5411.46460941 _step‚519 & lr-AdamW/pg1‚0.0003545017901250333 & lr-AdamW/pg2‚0.0003545017901250333 trainer/global_step‚6349²$)×˦Š_runtime‚ 5411.46460941 _step‚519 & lr-AdamW/pg1‚0.0003545017901250333 & lr-AdamW/pg2‚0.0003545017901250333 trainer/global_step‚6349 % _timestamp‚1.7843457527710369e+09‡øŽ%12§- _wandbruntime‚5411  _runtime‚5411¹Ûíûé'¨ãO trainer/global_step‚6349 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50277052 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109054103 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212635277027328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109044124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6705259158528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6704917819392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50277050 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚963959861862400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970665138060288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219410219729408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6705259158528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50277050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50277052 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58777051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚959007981375488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11124431 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚963959878901760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚965702661731886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6695020906288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58777051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970664779681792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212635258415104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219410339501568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963959861862400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6774961314304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6704917819392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219410219729408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6705259158528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50277050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50277052 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58777051 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚959007981375488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11124431 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚963959878901760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚965702661731886 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6695020906288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58777051 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970664779681792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212635258415104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219410339501568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963959861862400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6774961314304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6704917819392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6704917819392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50277050 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚963959861862400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970665138060288 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50284682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚964107102892032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109060808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24772816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6775980874752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843457533882823e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58776126 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965850183457480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219444785716224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚964107119931392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6706271792640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219444668048384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6776079930368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965850541046842 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58786117 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6705930446336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 trainer/global_step‚6349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50284684 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58776126 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚964107119931392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376bé!¬÷&¬ñM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219444785716224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚964107119931392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6706271792640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219444668048384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6776079930368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965850541046842 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58786117 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6705930446336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 trainer/global_step‚6349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50284684 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58776126 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚964107119931392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6696031960122 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50284682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚964107102892032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109060808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24772816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6775980874752 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843457533882823e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58776126 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965850183457480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59231503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214306696341504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978077885043712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973078811812394 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59221526 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚966333514969088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50658652 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚971321913344000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13748296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971321913344000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6826070845952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973079169401644 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50658650 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59231503 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221132880668160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11211678 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50658652 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978077526665216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978077526665216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6826165714432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155!ÂaE\ »çO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50658650 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚524 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6755613321216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6755954660352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843457932017639e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109880176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59221526 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221132767187456 SEDeviceStatsMonitor.on_train_batch_stÓ©8°‘art/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214306714953728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚5451.895493631 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6755954660352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚978077885043712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59231503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214306696341504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978077885043712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973078811812394 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59221526 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚966333514969088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50658652 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚971321913344000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13748296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971321913344000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6826070845952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973079169401644 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50658650 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59231503 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221132880668160 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11211678 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50658652 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978077526665216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978077526665216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6826165714432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6756967294464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226138707456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚978225780321792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6756625948160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221167332570112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6746307897028 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24964030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971469171412992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971469154373632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214340125100032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966480025640960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59230578 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971469154373632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6746648447030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6827188857856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214340143712256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24963772 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13750349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50666282 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6756625948160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6756967294464 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6827106579456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6827106579456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6756967294464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226138707456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚978225780321792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6756625948160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221167332570112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6746307897028 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24964030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971469171412992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971469154373632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214340125100032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966480025640960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59230578 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971469154373632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6746648447030 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6827188857856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214340143712256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24963772 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13750349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50666282 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6756625948160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088TâHH2¿- _wandbruntime‚5452  _runtime‚5452Äg7’ Àƒ %loss/total_loss‚8.752019637806683 trainer/global_step‚6399 epoch‚15 0loss/FRt1/assignment_loss‚1.6438471580278735 .loss/FRt2/detection_loss‚1.306893459980376 0loss/SRqqt2/detection_loss‚0.630588147851768 /loss/FBt1/detection_loss‚0.9535055216143115 /loss/FBt2/detection_loss‚0.6233810780162458 _runtime‚5452.513436536 _step‚526 2loss/SRqqt1/assignment_loss‚0.5816818378617178 2loss/FBt2/assignment_loss‚0.019682449574751005 1loss/SRqqt1/detection_loss‚1.0317700002922525 3loss/SRqqt2/assignment_loss‚0.48749391127569425 2loss/FBt1/assignment_loss‚0.006186946703806089 /loss/FRt1/detection_loss‚0.5962432729453971 % _timestamp‚1.7843457938201544e+09 0loss/FRt2/assignment_loss‚0.8707458536624908ó ‹¡Áˆ 3loss/SRqqt2/assignment_loss‚0.48749391127569425 2loss/FBt1/assignment_loss‚0.006186946703806089 /loss/FRt1/detection_loss‚0.5962432729453971 % _timestamp‚1.7843457938201544e+09 0loss/FRt2/assignment_loss‚0.8707458536624908 %loss/total_loss‚8.752019637806683 trainer/global_step‚6399 epoch‚15 0loss/FRt1/assignment_loss‚1.6438471580278735 .loss/FRt2/detection_loss‚1.306893459980376 0loss/SRqqt2/detection_loss‚0.630588147851768 /loss/FBt1/detection_loss‚0.9535055216143115 /loss/FBt2/detection_loss‚0.6233810780162458 _runtime‚5452.513436536 _step‚526 2loss/SRqqt1/assignment_loss‚0.5816818378617178 2loss/FBt2/assignment_loss‚0.019682449574751005 1loss/SRqqt1/detection_loss‚1.0317700002922525Žd:ºJžÂ‚(:“ ÉéëÒ“¯” gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.444799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò°3ØÂ(:Í ÉéëÒۿ͘ disk.sda.out‚ 177.17578125 proc.cpu.threads‚17 network.sent‚ 1.0073173e+07( disk.nvme0c0n1.out‚21223.68701171875 disk.dm-0.in‚11176.33203125# disk.dm-0.out‚21223.68701171875 disk.sda1.out‚0( proc.memory.availableMB‚ 496.61328125# proc.memory.rssMB‚ 2720.77734375 network.recv‚ 1.491929e+06) disk./.usagePercent‚40.25348954653652 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚11176.33203125 disk.rbd0.in‚0) proc.memory.percent‚16.60630702972412 disk.dm-1.in‚0 disk.dm-1.out‚ 177.17578125 disk.sda3.out‚ 177.17578125 disk.rbd1.out‚ 13.45703125$ disk./.usageGB‚599.8060836791992& disk.nvme0n1.out‚21223.68701171875# memory_percent‚96.9689130783081 cpu‚26.131268901867063 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚11176.33203125 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0•Ù,2Ä- _wandbruntime‚5470  _runtime‚5470ÚºvÏž{ 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.223858043551445 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10452229529619217 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6926201581954956 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.746647834777832 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961239099502563 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.89915931224823 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8276077508926392 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08826281875371933 /particle/accuracy_3_of_4‚0.9944444298744202 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7726825475692749 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7749738097190857 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22340795397758484 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6661646366119385 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚15 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027243146672844887 )jet/accuracy_1_of_3‚0.908793568611145 jet/accuracy_4_of_5‚0 7 Purity/*FRtµr´®50SRqqt*FBt/FBt_purity‚0.9962754845619202 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7455286383628845 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7804237604141235 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6609514951705933 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7146300077438354 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 /particle/accuracy_4_of_4‚0.9437788724899292 (particle/accuracy‚0.8516052961349487 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997619092464447 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7902451753616333 +particle/sensitivity‚0.6384070515632629 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48724302649497986 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984214186668396 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2629152834415436 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 'particle/f_score‚0.6488497853279114 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02570275217294693 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7099314332008362 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9099175333976746 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8771792650222778 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12284134328365326 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6956213712692261 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7551106214523315 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21193136274814606 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34624210000038147 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984408020973206 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891201853752136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5994299650192261 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6661646366119385 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10602043569087982 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2737058699131012 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09750000387430191 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7241877913475037 % _timestamp‚1.7843458119949737e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20376098155975342 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5170807838439941 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5021796226501465 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19994951784610748 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4791328012943268 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027243146672844887 particle/accuracy_3_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8320862054824829 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950022101402283 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6288119554519653 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7368372082710266 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 _step‚527 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684886813163757 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31983810663223267 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2629152834415436 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47605109214782715 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042285412549972534 *jet/accuracy_1_of_4‚0.9887303113937378 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895362854003906 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6747111082077026 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7573129534721375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20318596065044403 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22434209287166595 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06002405658364296 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_3_of_4‚0.1000269278883934 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05417557433247566 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7056431770324707 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2089207023382187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2296607345342636 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25975069403648376 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995887041091919 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6582605838775635 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 trainer/global_step‚6399 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7074078321456909 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02412366308271885 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8743788599967957 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.044408634305000305 +jet/accuracy_3_of_3‚0.07572950422763824 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8397204279899597 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2740173041820526 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7364442348480225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6947464346885681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6503069996833801 *jet/accuracy_1_of_2‚0.7710536122322083 *jet/accuracy_2_of_3‚0.5867802500724792 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2968750298023224 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7531922459602356 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.8092520833015442 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8252079486846924 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03159559145569801 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8657366633415222 (loss/val_total_loss‚8.01953368695246 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9388786554336548 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956476092338562 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6419281959533691 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18478277325630188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7182806730270386 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49843066930770874 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2100614756345749 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6418818235397339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6642770767211914 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24524791538715363 _runtime‚5470.689420087 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22152188420295715 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8657366633415222 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25352516770362854 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27160897850990295 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8380385637283325 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5679857730865479 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6713635325431824 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32487809658050537 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_1_of_1‚0.34611812233924866 6validation_average_jet_accuracy‚0.3835218846797943 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7600510120391846 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02478700876235962 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3109751343727112 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3952030539512634 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22536516189575195 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02614661306142807 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5697126388549805 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16375529766082764 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7459840178489685 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7905671000480652 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6284359693527222 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02335767075419426 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.398427814245224 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1’RMÉ=ÆÃ{ & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7551106214523315 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21193136274814606 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34624210000038147 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984408020973206 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891201853752136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5994299650192261 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6661646366119385 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10602043569087982 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2737058699131012 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09750000387430191 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7241877913475037 % _timestamp‚1.7843458119949737e+09 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20376098155975342 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5170807838439941 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5021796226501465 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19994951784610748 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4791328012943268 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027243146672844887 particle/accuracy_3_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8320862054824829 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950022101402283 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6288119554519653 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7368372082710266 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 _step‚527 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684886813163757 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31983810663223267 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2629152834415436 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47605109214782715 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042285412549972534 *jet/accuracy_1_of_4‚0.9887303113937378 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895362854003906 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6747111082077026 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7573129534721375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20318596065044403 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22434209287166595 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06002405658364296 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_3_of_4‚0.1000269278883934 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05417557433247566 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7056431770324707 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2089207023382187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2296607345342636 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25975069403648376 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995887041091919 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6582605838775635 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 trainer/global_step‚6399 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7074078321456909 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02412366308271885 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8743788599967957 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.044408634305000305 +jet/accuracy_3_of_3‚0.07572950422763824 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8397204279899597 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2740173041820526 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7364442348480225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6947464346885681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6503069996833801 *jet/accuracy_1_of_2‚0.7710536122322083 *jet/accuracy_2_of_3‚0.5867802500724792 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2968750298023224 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7531922459602356 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.8092520833015442 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8252079486846924 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03159559145569801 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8657366633415222 (loss/val_total_loss‚8.01953368695246 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9388786554336548 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956476092338562 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6419281959533691 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18478277325630188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7182806730270386 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49843066930770874 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2100614756345749 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6418818235397339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6642770767211914 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24524791538715363 _runtime‚5470.689420087 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22152188420295715 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8657366633415222 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25352516770362854 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27160897850990295 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8380385637283325 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5679857730865479 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6713635325431824 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32487809658050537 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_1_of_1‚0.34611812233924866 6validation_average_jet_accuracy‚0.3835218846797943 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7600510120391846 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02478700876235962 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3109751343727112 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3952030539512634 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22536516189575195 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02614661306142807 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5697126388549805 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16375529766082764 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7459840178489685 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7905671000480652 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6284359693527222 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02335767075419426 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.398427814245224 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.223858043551445 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10452229529619217 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6926201581954956 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.746647834777832 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961239099502563 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.89915931224823 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8276077508926392 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08826281875371933 /particle/accuracy_3_of_4‚0.9944444298744202 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7726825475692749 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7749738097190857 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22340795397758484 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6661646366119385 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚15 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027243146672844887 )jet/accuracy_1_of_3‚0.908793568611145 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962754845619202 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7455286383628845 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7804237604141235 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6609514951705933 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7146300077438354 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 /particle/accuracy_4_of_4‚0.9437788724899292 (particle/accuracy‚0.8516052961349487 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997619092464447 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7902451753616333 +particle/sensitivity‚0.6384070515632629 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48724302649497986 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984214186668396 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2629152834415436 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 'particle/f_score‚0.6488497853279114 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02570275217294693 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7099314332008362 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9099175333976746 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8771792650222778 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12284134328365326 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6956213712692261 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0·‘µM“Ç‚(:ˆ ØéëÒ’Ø” gpu.0.gpu‚71.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š]Š†ÙÈ‚(:Î ØéëÒ¾Õ£– disk.sda1.out‚0$ memory_percent‚96.97461128234863 disk.rbd1.in‚0' proc.memory.availableMB‚ 495.6796875 network.sent‚ 1.0122997e+07 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚11176.33203125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚21228.98486328125 disk.sda3.in‚0 disk.sda.out‚ 177.55859375& disk.nvme0n1.out‚21228.98486328125) proc.memory.percent‚16.60630702972412 cpu‚26.139110360796206 proc.cpu.threads‚17 disk.rbd1.out‚ 13.51953125 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚21228.98486328125* disk./.usagePercent‚40.253533323824875$ disk./.usageGB‚599.8067359924316 disk.dm-1.out‚ 177.55859375 disk.drbd2165.in‚0 disk.dm-0.in‚11176.33203125 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.sda.in‚0 network.recv‚ 1.496682e+06 disk.sda3.out‚ 177.55859375 disk.sda2.in‚0" disk.nvme0n1.in‚11176.33203125(;ç“É‚(:ˆ çéëÒÅñæ“ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0![qÆÕÊ‚(:Ê çéëÒž‘Þ– disk.drbd2165.out‚0# proc.memory.rssMB‚ 2720.77734375) disk./.usagePercent‚40.25352641162145 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 11177.1640625# disk.dm-0.out‚21235.90966796875 disk.sda.out‚ 177.78515625 disk.sda.in‚0! disk.nvme0n1.in‚ 11177.1640625 disk.sda3.in‚0 disk.sda3.out‚ 177.78515625) proc.memory.percent‚16.60630702972412 disk.rbd0.out‚0 disk.dm-1.out‚ 177.78515625 disk.sda2.in‚0 network.recv‚ 1.501592e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 11177.1640625( disk.nvme0c0n1.out‚21235.90966796875 disk.sda2.out‚0& disk.nvme0n1.out‚21235.90966796875$ memory_percent‚96.97403907775879$ disk./.usageGB‚599.8066329956055 disk.sda1.in‚0 disk.rbd1.out‚ 13.51953125' proc.memory.availableMB‚ 495.7734375 network.sent‚ 1.0180462e+07 disk.rbd0.in‚0 disk.rbd1.in‚0 cpu‚26.139724363248483 proc.cpu.threads‚17\üÙ÷„Ë‚(:“ öéëÒõ¡µ“ gpu.0.gpu‚46.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.093' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.818600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uŽEà0ncorrectedMemoryErrors‚0j ÐÌ‚(:Å öéëÒ»£õ• disk.sda2.in‚0" disk.dm-0.out‚21244.9814453125 disk.sda.in‚0% disk.nvme0n1.out‚21244.9814453125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 13.64453125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚21244.9814453125 proc.cpu.threads‚17* disk./.usagePercent‚40.296246132852595 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚96.97418212890625 network.sent‚ 1.0187375e+07 network.recv‚ 1.504939e+06# disk./.usageGB‚600.443187713623# proc.memory.rssMB‚ 2720.77734375) proc.memory.percent‚16.60630702972412 cpu‚26.140347584315872 disk.sda3.out‚ 178.13671875 disk.rbd0.out‚0 disk.dm-0.in‚11293.30078125" proc.memory.availableMB‚495.75" disk.nvme0n1.in‚11293.30078125 disk.dm-1.out‚ 178.13671875$ disk.nvme0c0n1.in‚11293.30078125 disk.sda.out‚ 178.13671875*{ë´2Í- _wandbruntime‚5510  _runtime‚5510W à€ÇÎÁ & lr-AdamW/pg1‚0.0003515503333885954 & lr-AdamW/pg2‚0.0003515503333885954 trainer/global_step‚6449 % _timestamp‚1.7843458519698596e+09 _runtime‚5510.663490463 _step‚528X$íÌÏÆ % _timestamp‚1.7843458519698596e+09 _runtime‚5510.663490463 _step‚528 & lr-AdamW/pg1‚0.0003515503333885954 & lr-AdamW/pg2‚0.0003515503333885954 trainer/global_step‚6449ÕSR2Ð- _wandbruntime‚5510  _runtime‚5510=nøç'ÑáO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51160918 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6813973605333 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51160916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚980613531475968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222889626601984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975567711776768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59746300 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215994064979968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚980613531475968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚980613548515328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987437931883008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6449 _step‚529 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982381344832211 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110917195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51160916 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6824383367680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13883797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59756277 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚982381702421461 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚975567728816128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843458519709964e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110907216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987437573504512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59746300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6895479336448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987437573504512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25185248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987437931883008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51160918 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51160918 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6813973605333 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51160916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚980613531475968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222889626601984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚975567711776768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59746300 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215994064979968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚980613531475968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚980613548515328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987437931883008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6449 _step‚529 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982381344832211 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110917195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51160916 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6824383367680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13883797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59756277 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚982381702421461 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚975567728816128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843458519709964e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110907216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987437573504512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59746300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6895479336448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987437573504512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25185248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987437931883008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51160918 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25189086 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222924084690944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982528866557805 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6825396001792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 trainer/global_step‚6449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6814644109165 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6814984659167 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6825396001792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚980760772505600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987585827161088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222923977508864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110923900 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13885890 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25188832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6825054655488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975714222448640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51168550 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11303193 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933893 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110933893 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110923900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51168548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6896590952448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843458525849366e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51168548 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6896502382592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6825054655488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987586185546752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216027475126272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982529224147167 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚980760772505600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987586185546752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216027493738496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51168550 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11303196 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59755352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25189086 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222924084690944 ;5DeviceStatsMonitor.ongS¿m$_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982528866557805 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6825396001792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 trainer/global_step‚6449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6814644109165 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6814984659167 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6825396001792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚980760772505600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987585827161088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222923977508864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110923900 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13885890 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25188832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6825054655488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975714222448640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51168550 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11303193 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933893 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110933893 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110923900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51168548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6896590952448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843458525849366e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51168548 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6896502382592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6825054655488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987586185546752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216027475126272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982529224147167 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚980760772505600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987586185546752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216027493738496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51168550 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11303196 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59755352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EŸ2ß- _wandbruntime‚5550  _runtime‚5550g>ÞGÇàÁ trainer/global_step‚6499 % _timestamp‚1.7843458922947276e+09 _runtime‚5550.988298675 _step‚532 & lr-AdamW/pg1‚0.0003500623208943849 & lr-AdamW/pg2‚0.0003500623208943849žÆ9×ÌáÆ _runtime‚5550.988298675 _step‚532 & lr-AdamW/pg1‚0.0003500623208943849 & lr-AdamW/pg2‚0.0003500623208943849 trainer/global_step‚6499 % _timestamp‚1.7843458922947276e+09”˨ý“2â- _wandbruntime‚5550  _runtime‚5550"hËËè'ãâO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987975582957568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987975582957568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6946970403840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚5550.989517405 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11388701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60200752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217665502906368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6864590092241 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13987707 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843458922958076e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994850320487936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994850678866432 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚987975599996928 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51542518 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60210729 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224612557692928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60200752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994850320487936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217665484294144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6874737530368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25376408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508Ñ[4äçO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11388701 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60200752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217665502906368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6864590092241 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13987707 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843458922958076e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994850320487936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994850678866432 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚987975599996928 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51542518 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60210729 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224612557692928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60200752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994850320487936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217665484294144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6874737530368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25376408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987975582957568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987975582957568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6946970403840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚5550.989517405•àb÷›2å- _wandbruntime‚5551  _runtime‚5551&,íñ&æëM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13989504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25379947 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51550150 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51550148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚994998932530176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚988122823987200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6876091503616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚988122841026560 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚5551.604540075 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60219795 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217698931664896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983039756042240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6948088073728 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60209804 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25380197 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚994998932530176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚994998574144512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13989751 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224646929333760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60209804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843458929106646e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6948016281088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚989905374227675 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983039773081600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6875750157312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6499 _step‚534 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚988122823987200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚989905016638313 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11390443 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6865601146075 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6875750157312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51550148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚988122841026560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769945YÞxö&çðM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111769945 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6499 _step‚534 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚988122823987200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚989905016638313 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11390443 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6865601146075 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6875750157312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51550148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚988122841026560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13989504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25379947 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51550150 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51550148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚994998932530176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚988122823987200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6876091503616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚988122841026560 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚5551.604540075 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60219795 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217698931664896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983039756042240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6948088073728 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60209804 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25380197 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚994998932530176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚994998574144512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13989751 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224646929333760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60209804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843458929106646e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6948016281088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚989905374227675 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983039773081600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6875750157312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002263067471360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚990218796003328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1002263425849856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1002263425849856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5591.538676268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002263067471360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚995337651478528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51924118 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51924118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51924116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14091429 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6915206579149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112579320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚995337634439168 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6925774371328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226335248433152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60655204 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25567641 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60655204 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14091690 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843459328449333e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚995337634439168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6925774371328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6914866029259 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60665181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60665181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11475948 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6998307600384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚997133644993227 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 „x5ñ'ôëO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226335145438208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚219336922220544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5591.538676268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002263067471360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚995337651478528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51924118 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51924118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51924116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14091429 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6915206579149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112579320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚995337634439168 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6925774371328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226335248433152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60655204 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25567641 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60655204 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14091690 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843459328449333e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚995337634439168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6925774371328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6914866029259 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60665181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60665181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11475948 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6998307600384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚997133644993227 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6925433032192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112589299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002263067471360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚990218796003328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1002263425849856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1002263425849856 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25571199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60664256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚990365289635840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25571443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14093506 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚990365306675200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112596004 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60664256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚995484892508160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219370369591296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916217632983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6926445659136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112596004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112605997 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6999256208896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚997281166718821 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚995484892508160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60674247 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚997281524308183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51931750 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51931748 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60674247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11477696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚995484875468800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843459334580336e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002411679513600 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002411321127936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219370350979072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ê´0Ê€÷ôM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚995484892508160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60674247 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚997281524308183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51931750 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51931748 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60674247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11477696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚995484875468800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843459334580336e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002411679513600 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002411321127936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219370350979072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14093747 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226369705981440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6926445659136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25571199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60664256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚990365289635840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25571443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14093506 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚990365306675200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112596004 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60664256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚995484892508160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219370369591296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916217632983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6926445659136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112596004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112605997 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6999256208896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚997281166718821 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚538 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376š®Ð2ø- _wandbruntime‚5592  _runtime‚5592Yk¢Ëù€ /loss/FRt1/assignment_loss‚1.747272470908333 2loss/SRqqt1/assignment_loss‚0.6039724264309352 2loss/FBt1/assignment_loss‚0.012322993947582007 /loss/FRt1/detection_loss‚0.6216369975417294 0loss/FRt2/assignment_loss‚0.7934825978717025 /loss/FRt2/detection_loss‚1.2690093066501764 0loss/SRqqt1/detection_loss‚1.024684932304635 .loss/FBt1/detection_loss‚1.063590801526637 /loss/FBt2/detection_loss‚0.5083672533384882 epoch‚16 _runtime‚5592.154288505 1loss/FBt2/assignment_loss‚0.03304396470462309 1loss/SRqqt2/detection_loss‚0.5869270977648312 %loss/total_loss‚8.767691500497445 trainer/global_step‚6549 % _timestamp‚1.7843459334609563e+09 _step‚539 2loss/SRqqt2/assignment_loss‚0.5033806575077736} ÷· ú… 2loss/SRqqt2/assignment_loss‚0.5033806575077736 /loss/FRt1/assignment_loss‚1.747272470908333 2loss/SRqqt1/assignment_loss‚0.6039724264309352 2loss/FBt1/assignment_loss‚0.012322993947582007 /loss/FRt1/detection_loss‚0.6216369975417294 0loss/FRt2/assignment_loss‚0.7934825978717025 /loss/FRt2/detection_loss‚1.2690093066501764 0loss/SRqqt1/detection_loss‚1.024684932304635 .loss/FBt1/detection_loss‚1.063590801526637 /loss/FBt2/detection_loss‚0.5083672533384882 epoch‚16 _runtime‚5592.154288505 1loss/FBt2/assignment_loss‚0.03304396470462309 1loss/SRqqt2/detection_loss‚0.5869270977648312 %loss/total_loss‚8.767691500497445 trainer/global_step‚6549 % _timestamp‚1.7843459334609563e+09 _step‚539›ÌjAD‘û‚(:† ÐêëÒÓ’¢” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚251.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.’òÚÖü‚(:Ë ÐêëÒ§³˜– cpu‚26.144055713735835 network.recv‚ 1.528558e+06) disk./.usagePercent‚40.25181551327012 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 495.28515625 proc.cpu.threads‚17 disk.rbd1.out‚ 13.83203125 disk.sda.in‚0% disk.nvme0n1.out‚22104.4580078125 disk.sda3.out‚ 180.43359375$ disk./.usageGB‚599.7811393737793 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 network.sent‚ 1.0331268e+07 disk.rbd1.in‚0 disk.dm-1.out‚ 180.43359375" disk.nvme0n1.in‚11452.48046875) proc.memory.percent‚16.60630702972412 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚11452.48046875' disk.nvme0c0n1.out‚22104.4580078125 disk.dm-0.in‚11452.48046875" disk.dm-0.out‚22104.4580078125$ memory_percent‚96.97701930999756 disk.sda.out‚ 180.43359375 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.77734375ÈŸ/Øžý‚(:“ ßêëÒÚúÞ” gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.629' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.925800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôŽÚ°Øþ‚(:Í ßêëÒºÞð• disk.sda1.out‚0 disk.dm-1.out‚ 180.65234375( proc.memory.availableMB‚ 495.61328125 disk.dm-0.in‚11452.48046875 disk.sda.out‚ 180.65234375 disk.sda2.in‚0) proc.memory.percent‚16.60630702972412( disk.nvme0c0n1.out‚22109.20458984375# disk.dm-0.out‚22109.20458984375& disk.nvme0n1.out‚22109.20458984375 disk.sda3.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚11452.48046875 disk.sda3.out‚ 180.65234375 disk.rbd1.out‚ 13.83203125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.sda.in‚0" disk.nvme0n1.in‚11452.48046875$ disk./.usageGB‚599.7811393737793 proc.cpu.threads‚17 network.sent‚ 1.0394316e+07) disk./.usagePercent‚40.25181551327012 disk.rbd0.out‚0 disk.drbd2165.out‚0# memory_percent‚96.9750165939331 network.recv‚ 1.533333e+06 disk.sda2.out‚0 disk.drbd2165.in‚0 cpu‚26.144606581679103Šn5Š‘ÿ‚(:† îêëÒ©ÙÝ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.488' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚™É€‚(:¾ îêëÒ´þº–$ disk./.usageGB‚599.7813186645508 disk.sda.out‚ 180.94921875 disk.sda2.out‚0 disk.dm-0.in‚ 11453.3125) proc.memory.percent‚16.60630702972412 disk.sda2.in‚0 disk.nvme0n1.in‚ 11453.3125' proc.memory.availableMB‚ 495.5703125 network.recv‚ 1.536557e+06 disk.sda3.out‚ 180.94921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0) disk./.usagePercent‚40.25182754562423 disk.rbd1.in‚0 disk.dm-1.out‚ 180.94921875 disk.nvme0c0n1.in‚ 11453.3125" disk.dm-0.out‚22121.3896484375 proc.cpu.threads‚17 disk.sda1.out‚0% disk.nvme0n1.out‚22121.3896484375 disk.sda.in‚0# proc.memory.rssMB‚ 2720.77734375 cpu‚26.145278608421393 disk.dm-1.in‚0 network.sent‚ 1.0399156e+07 disk.sda3.in‚0 disk.rbd1.out‚ 13.89453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚22121.3896484375$ memory_percent‚96.97527885437012fLŽˆ2- _wandbruntime‚5631  _runtime‚5631jB° Ç‚Á trainer/global_step‚6599 % _timestamp‚1.7843459732428062e+09 _runtime‚5631.936378954 _step‚540 & lr-AdamW/pg1‚0.0003470622055091521 & lr-AdamW/pg2‚0.00034706220550915217þƒ}ÌƒÆ & lr-AdamW/pg1‚0.0003470622055091521 & lr-AdamW/pg2‚0.0003470622055091521 trainer/global_step‚6599 % _timestamp‚1.7843459732428062e+09 _runtime‚5631.936378954 _step‚540œúÕ%W2„- _wandbruntime‚5631  _runtime‚5631•AÊð'…êO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002699702960128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25758192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002699685920768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61119633 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002699702960128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6976128534016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11563201 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11563198 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚997544312557568 trainer/global_step‚6599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009676172833280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228057821543936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6976469873152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52305718 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7049535293952 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1004510152662985 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52305716 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6976469873152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7049461396992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221008378759168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14195248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7843459732438889e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009675814454784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚997544329596928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221008360146944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113425351 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25758449 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890ž?älõ'†ïO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009675814454784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1004509795073735 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14195248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7843459732438889e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009675814454784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚997544329596928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221008360146944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113425351 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25758449 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002699702960128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25758192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002699685920768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚541 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61119633 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002699702960128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6976128534016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11563201 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11563198 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚997544312557568 trainer/global_step‚6599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009676172833280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228057821543936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6976469873152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52305718 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7049535293952 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1004510152662985 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52305716 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6976469873152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7049461396992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221008378759168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109656 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25761908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all*{$†|.freed‚228092254433280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25762180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221041807517696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6977141160960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52313348 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61118708 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966493569889 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002846926950400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6977141160960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113442049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6966834119891 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1002846943989760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113432056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52313350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61118708 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113432056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11564946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚6599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52313350 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009824426497024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61128699 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1009824068111360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228092370003968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442049 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843459738606808e+09ü".ø'‰ûM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52313348 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61118708 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966493569889 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002846926950400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6977141160960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113442049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6966834119891 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1002846943989760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113432056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52313350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61118708 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113432056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11564946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚6599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52313350 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009824426497024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61128699 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1009824068111360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228092370003968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442049 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843459738606808e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61128699 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6977482507264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _runtime‚5632.554463155 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1004657674388691 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002846943989760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11564943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221041788905472 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25761908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228092254433280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25762180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221041807517696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6977141160960ž|JPU2Š- _wandbruntime‚5632  _runtime‚5632•fú‹‚ 1loss/SRqqt2/detection_loss‚0.6789443743094054 epoch‚16 % _timestamp‚1.7843459738639507e+09 _step‚543 0loss/FRt2/assignment_loss‚0.8235940711442814 2loss/SRqqt1/assignment_loss‚0.5424064908024792 2loss/FBt1/assignment_loss‚0.003121232468621897 2loss/FBt2/assignment_loss‚0.011023324664493541 /loss/FRt1/detection_loss‚0.6200742943424776 /loss/FBt1/detection_loss‚0.6533784126560833 %loss/total_loss‚8.140666829264111 trainer/global_step‚6599 0loss/FBt2/detection_loss‚0.40545917100260137 _runtime‚5632.557243834 /loss/FRt1/assignment_loss‚1.641947420500219 1loss/SRqqt2/assignment_loss‚0.527292608100333 /loss/FRt2/detection_loss‚1.2602412968258183 0loss/SRqqt1/detection_loss‚0.973184132447297·ò Œ‡ 2loss/SRqqt1/assignment_loss‚0.5424064908024792 2loss/FBt1/assignment_loss‚0.003121232468621897 2loss/FBt2/assignment_loss‚0.011023324664493541 /loss/FRt1/detection_loss‚0.6200742943424776 /loss/FBt1/detection_loss‚0.6533784126560833 %loss/total_loss‚8.140666829264111 trainer/global_step‚6599 0loss/FBt2/detection_loss‚0.40545917100260137 _runtime‚5632.557243834 /loss/FRt1/assignment_loss‚1.641947420500219 1loss/SRqqt2/assignment_loss‚0.527292608100333 /loss/FRt2/detection_loss‚1.2602412968258183 0loss/SRqqt1/detection_loss‚0.973184132447297 1loss/SRqqt2/detection_loss‚0.6789443743094054 epoch‚16 % _timestamp‚1.7843459738639507e+09 _step‚543 0loss/FRt2/assignment_loss‚0.8235940711442814Ÿ‰`nýž‚(:“ ýêëÒàý– gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.094' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.218799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 w“ÔŽ‚(:É ýêëÒƒž—( disk./.usagePercent‚40.2518190973756 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 181.35546875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 11524.0859375! disk.nvme0n1.in‚ 11524.0859375& disk.nvme0n1.out‚22124.58544921875# proc.memory.rssMB‚ 2720.77734375 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚22124.58544921875$ memory_percent‚96.98007106781006 cpu‚26.145724688522442 network.sent‚ 1.0404223e+07 disk.dm-1.out‚ 181.35546875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 11524.0859375( disk.nvme0c0n1.out‚22124.58544921875( proc.memory.availableMB‚ 494.78515625) proc.memory.percent‚16.60630702972412 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.out‚ 181.35546875 disk.sda2.out‚0 network.recv‚ 1.540355e+06# disk./.usageGB‚599.781192779541 disk.rbd1.out‚ 13.89453125 disk.rbd0.out‚0 disk.drbd2165.in‚0­ßèž‚(:“ ŒëëÒ÷ÞÓ“ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.515800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚042Lè‚(:· ŒëëÒÙÙ·• network.sent‚ 1.0467286e+07 network.recv‚ 1.545299e+06 disk.sda1.in‚0 disk.dm-0.in‚11525.05078125 disk.rbd1.out‚ 14.01953125# memory_percent‚96.9757080078125 disk.rbd0.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2720.77734375 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚22134.056640625 disk.sda.in‚0 disk.sda2.out‚0$ disk./.usageGB‚599.7812194824219 disk.sda1.out‚0& disk.nvme0c0n1.out‚22134.056640625 disk.sda2.in‚0$ disk.nvme0n1.out‚22134.056640625 disk.sda3.out‚ 181.71875* disk./.usagePercent‚40.251820889428345 disk.drbd2165.in‚0 disk.rbd1.in‚0! proc.memory.availableMB‚495.5 disk.dm-1.out‚ 181.71875 disk.sda.out‚ 181.71875" disk.nvme0n1.in‚11525.05078125) proc.memory.percent‚16.60630702972412 cpu‚26.14628100748099$ disk.nvme0c0n1.in‚11525.05078125 disk.sda3.in‚0 proc.cpu.threads‚17@–“‘‚(:ˆ ›ëëÒãàÍ“ gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹0ÚÊ’‚(:¿ ›ëëÒãµð• disk.sda2.in‚0 disk.dm-0.in‚ 11525.8828125 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚22141.05859375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-0.out‚22141.05859375) proc.memory.percent‚16.60630702972412 disk.drbd2165.out‚0 disk.sda.out‚ 181.9453125$ memory_percent‚96.97740077972412 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 11525.8828125% disk.nvme0c0n1.out‚22141.05859375 network.sent‚ 1.0472243e+07! disk.nvme0n1.in‚ 11525.8828125 disk.dm-1.in‚0 disk.dm-1.out‚ 181.9453125( proc.memory.availableMB‚ 495.22265625 proc.cpu.threads‚17$ disk./.usageGB‚599.7814140319824 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 14.01953125 disk.rbd0.out‚0 disk.drbd2165.in‚0 network.recv‚ 1.548819e+06* disk./.usagePercent‚40.251833945812585 disk.sda3.out‚ 181.9453125# proc.memory.rssMB‚ 2720.77734375 cpu‚26.14672524428729³Å•2“- _wandbruntime‚5672  _runtime‚5672â¹ÞZÆ”À & lr-AdamW/pg1‚0.0003455502953099346 & lr-AdamW/pg2‚0.0003455502953099346 trainer/global_step‚6649 % _timestamp‚1.7843460137996762e+09 _runtime‚ 5672.49329291 _step‚544ï™ÂË•Å _runtime‚ 5672.49329291 _step‚544 & lr-AdamW/pg1‚0.0003455502953099346 & lr-AdamW/pg2‚0.0003455502953099346 trainer/global_step‚6649 % _timestamp‚1.7843460137996762e+09 /ŒnJ2–- _wandbruntime‚5672  _runtime‚5672òmNè¨#—îO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222679816685568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52687318 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61574085 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11650451 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11650448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114251424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1004869846151168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61564108 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1011885945154243 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52687316 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 trainer/global_step‚6649 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61574085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229780480997376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7026824035840 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010061754441728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviaãOÉLceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017088561438208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7026824035840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52687316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14298619 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25949067 _runtime‚5672.494351418 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7100769403904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7100682924032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3©mðõù'˜óO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7100769403904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7100682924032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222679816685568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52687318 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61574085 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11650451 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11650448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114251424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1004869846151168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61564108 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1011885945154243 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52687316 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 trainer/global_step‚6649 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61574085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229780480997376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7026824035840 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010061754441728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017088561438208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7026824035840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52687316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14298619 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25949067 _runtime‚5672.494351418¡+ƒã2™- _wandbruntime‚5673  _runtime‚5673EãþÔþ&šøM _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1010208995471360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114268108 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017237173480448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229815036412416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010208978432000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52694950 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222713226831872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25952847 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010208995471360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7017110056797 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61583151 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _step‚546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268108 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784346014415111e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25953119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52694950 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61573160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7017450606799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7028178009088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11652196 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7027836662784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229814916647424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222713245444096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7101689815552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61573160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52694948 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11652193 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012033466879837 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1005016373862400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017237173480448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7028178009088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114268108 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017237173480448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229815036412416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010208978432000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52694950 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222713226831872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25952847 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010208995471360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7017110056797 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61583151 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _step‚546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268108 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784346014415111e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25953119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52694950 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61573160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7017450606799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7028178009088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11652196 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7027836662784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229814916647424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222713245444096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7101689815552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61573160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52694948 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11652193 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012033466879837 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1005016373862400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017237173480448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7028178009088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231503304999424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚549 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62018560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1012195396784128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843460543488486e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017423788883968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017423788883968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚5713.042587745 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024501666800128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7077519537664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7151959713280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024501308421632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53068918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224351254611968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53068918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62028537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1019262452824001 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7067056039873 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7077860876800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14402196 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62018560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26140177 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224351235999744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11737698 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231503195713024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1019262095234751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017423805923328 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1024501308421632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53068916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14402476 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7066715489983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088Œ´Sžù'¨óO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224351235999744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11737698 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231503195713024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1019262095234751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017423805923328 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1024501308421632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53068916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14402476 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7066715489983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7077860876800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017423805923328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62028537 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚6699 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53068916 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231503304999424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚549 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62018560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1012195396784128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843460543488486e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017423788883968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017423788883968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚5713.042587745 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024501666800128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7077519537664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7151959713280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024501308421632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53068918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224351254611968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53068918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62028537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1019262452824001 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7067056039873 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7077860876800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14402196 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62018560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26140177 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976¥kå¿'2©- _wandbruntime‚5713  _runtime‚5713œ]2Ì!ªøM KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚550 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53076548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024649920463872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017571029913600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024649920463872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115104160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53076550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5713.657121121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53076550 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53076548 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7078873510912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62037603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26143717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231537753458688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚6699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017571046952960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26143983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017571046952960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7078532164608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1012341890416640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62037603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14404274 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7078873510912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11739443 % _timestamp‚1.7843460549633403e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14404537 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591Šî%~'«ýM U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024649920463872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017571029913600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024649920463872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115104160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53076550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5713.657121121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53076550 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53076548 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7078873510912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62037603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26143717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231537753458688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚6699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017571046952960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26143983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017571046952960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7078532164608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1012341890416640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62037603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14404274 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7078873510912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11739443 % _timestamp‚1.7843460549633403e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14404537 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚550 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53076548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888¦÷|ø2¬- _wandbruntime‚5713  _runtime‚5713wÿ]¾ ­… 2loss/SRqqt1/assignment_loss‚0.6018594055409014 2loss/SRqqt2/assignment_loss‚0.4565963704850462 2loss/FBt2/assignment_loss‚0.005815468842295828 1loss/SRqqt2/detection_loss‚0.5834129523732985 /loss/FBt1/detection_loss‚0.6411681414936625 /loss/FBt2/detection_loss‚0.3809524377348781 trainer/global_step‚6699 0loss/FRt1/assignment_loss‚1.6450851677196112 3loss/FBt1/assignment_loss‚0.0007188725668206143 &loss/total_loss‚7.8484935058172915 0loss/FRt2/assignment_loss‚0.8715818930864334 % _timestamp‚1.7843460549663095e+09 _step‚551 /loss/FRt1/detection_loss‚0.6255076190585555 .loss/FRt2/detection_loss‚1.296618893084582 1loss/SRqqt1/detection_loss‚0.7391762838312067 epoch‚16 _runtime‚5713.659603357ÞAÍw®Š 0loss/FRt2/assignment_loss‚0.8715818930864334 % _timestamp‚1.7843460549663095e+09 _step‚551 /loss/FRt1/detection_loss‚0.6255076190585555 .loss/FRt2/detection_loss‚1.296618893084582 1loss/SRqqt1/detection_loss‚0.7391762838312067 epoch‚16 _runtime‚5713.659603357 2loss/SRqqt1/assignment_loss‚0.6018594055409014 2loss/SRqqt2/assignment_loss‚0.4565963704850462 2loss/FBt2/assignment_loss‚0.005815468842295828 1loss/SRqqt2/detection_loss‚0.5834129523732985 /loss/FBt1/detection_loss‚0.6411681414936625 /loss/FBt2/detection_loss‚0.3809524377348781 trainer/global_step‚6699 0loss/FRt1/assignment_loss‚1.6450851677196112 3loss/FBt1/assignment_loss‚0.0007188725668206143 &loss/total_loss‚7.8484935058172915§t´—Ư‚(:» ÈëëÒ³µ–! disk.nvme0c0n1.in‚ 11619.71875% disk.nvme0c0n1.out‚22161.40234375 disk.dm-0.out‚22161.40234375 network.recv‚ 1.561227e+06 network.sent‚ 1.054511e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.sda2.in‚0 disk.sda3.out‚ 183.09765625 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚599.7808265686035 disk.sda1.out‚0 disk.dm-1.out‚ 183.09765625 disk.dm-0.in‚ 11619.71875 disk.rbd1.in‚0 disk.rbd1.out‚ 14.14453125 disk.sda.out‚ 183.09765625 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚96.97940349578857) disk./.usagePercent‚40.25179452065231 disk.sda2.out‚0 disk.nvme0n1.in‚ 11619.71875# disk.nvme0n1.out‚22161.40234375( proc.memory.availableMB‚ 494.89453125) proc.memory.percent‚16.60630702972412 cpu‚26.148312690774063 disk.sda3.in‚0 disk.dm-1.in‚0lÃÁá“°‚(:ˆ ÈëëÒㆷ› gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V^´ž±‚(:’ ×ëëÒЋ¥” gpu.0.gpu‚15.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.093' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.818600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=]¢ÎȲ‚(:½ ×ëëÒݲ½– network.sent‚ 1.0608206e+07) proc.memory.percent‚16.60630702972412 disk.drbd2165.out‚0( proc.memory.availableMB‚ 495.05078125* disk./.usagePercent‚40.251792728599575 disk.dm-1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚22165.36376953125 disk.nvme0n1.in‚ 11619.71875 disk.sda3.in‚0# proc.memory.rssMB‚ 2720.77734375 disk.dm-1.out‚ 183.21875! disk.nvme0c0n1.in‚ 11619.71875 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚96.97844982147217 disk.rbd1.in‚0 disk.rbd1.out‚ 14.14453125 disk.rbd0.out‚0 network.recv‚ 1.566252e+06 disk.dm-0.in‚ 11619.71875# disk.dm-0.out‚22165.36376953125 disk.sda.out‚ 183.21875 cpu‚26.148837482784845 disk.sda1.out‚0$ disk./.usageGB‚599.7807998657227 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚22165.36376953125 disk.sda3.out‚ 183.21875 proc.cpu.threads‚17$™Žž³‚(:“ æëëÒóðÁ“ gpu.0.gpu‚62.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚234.352' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.870400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"+HƒØ´‚(:Í æëëÒÖñו disk.drbd2165.in‚0 network.recv‚ 1.569845e+06 disk.dm-0.in‚11620.68359375& disk.nvme0n1.out‚22172.58154296875 disk.rbd1.out‚ 14.20703125# proc.memory.rssMB‚ 2720.77734375) disk./.usagePercent‚40.25172923873108" disk.nvme0n1.in‚11620.68359375 disk.dm-1.out‚ 183.61328125( proc.memory.availableMB‚ 494.78515625 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚22172.58154296875 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚96.98007106781006) proc.memory.percent‚16.60630702972412 cpu‚26.14941085696446 network.sent‚ 1.0613134e+07$ disk./.usageGB‚599.7798538208008 disk.sda2.in‚0$ disk.nvme0c0n1.in‚11620.68359375 proc.cpu.threads‚17 disk.sda3.out‚ 183.61328125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚22172.58154296875 disk.sda.in‚0 disk.sda.out‚ 183.61328125 disk.rbd1.in‚08#82µ- _wandbruntime‚5753  _runtime‚5753’XeǶÁ & lr-AdamW/pg1‚0.0003425032554278953 & lr-AdamW/pg2‚0.0003425032554278953 trainer/global_step‚6749 % _timestamp‚1.7843460947306802e+09 _runtime‚5753.424276556 _step‚552áfÒÞÌ·Æ & lr-AdamW/pg2‚0.0003425032554278953 trainer/global_step‚6749 % _timestamp‚1.7843460947306802e+09 _runtime‚5753.424276556 _step‚552 & lr-AdamW/pg1‚0.0003425032554278953¨º×ÍF2¸- _wandbruntime‚5753  _runtime‚5753ϳ)mô'¹îO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14504947 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031914055405056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53450518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226022673926144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026638602904509 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019520913338368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233225684570624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11824951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233225793857024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933507 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26330174 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024785857404928 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1024785840365568 _runtime‚5753.428374283 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7128215039488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1026638245315259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7117672526781 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128215039488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7203010644480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024785857404928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226022692538368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7117331976891 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53450518 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115923528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11824948 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1031914413783552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1024785840365568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7203101318656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1031914055405056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62473012 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53450516 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26329895 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269H{3¢èºóO aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024785857404928 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1024785840365568 _runtime‚5753.428374283 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7128215039488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1026638245315259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7117672526781 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128215039488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7203010644480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024785857404928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226022692538368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7117331976891 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53450518 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115923528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11824948 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1031914413783552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1024785840365568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7203101318656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1031914055405056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62473012 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53450516 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26329895 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14504947 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031914055405056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53450518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226022673926144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026638602904509 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019520913338368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233225684570624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11824951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233225793857024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933507 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26330174©þ“ts2»- _wandbruntime‚5754  _runtime‚5754B0Yþ&¼øM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1032062667447296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14507025 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115940212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14507274 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233260134417408 trainer/global_step‚6749 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53458150 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1026785767040853 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115940212 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024933098434560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129227666432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62492055 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11826693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53458148 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032062667447296 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115950205 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53458150 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1024933081395200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7118683580615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26333718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024933081395200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129569012736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1019667424010240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129227666432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226056121296896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1019667441049600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62482064 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1024933098434560 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1032062309061632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7204122399744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129569012736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026786124630215 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚5754.041460753 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664Î]Ö'½ýM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026786124630215 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚5754.041460753 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1032062667447296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14507025 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115940212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14507274 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233260134417408 trainer/global_step‚6749 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53458150 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1026785767040853 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115940212 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024933098434560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129227666432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62492055 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11826693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53458148 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032062667447296 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115950205 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53458150 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1024933081395200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7118683580615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26333718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024933081395200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129569012736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1019667424010240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129227666432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226056121296896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1019667441049600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62482064 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1024933098434560 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1032062309061632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7204122399744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129569012736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983ªb ߬2¾- _wandbruntime‚5754  _runtime‚5754FÚ©=¿ /loss/FRt1/assignment_loss‚1.734549213457724 0loss/FRt2/assignment_loss‚0.8328265684697138 3loss/FBt1/assignment_loss‚8.539289535036687e-06 /loss/FRt1/detection_loss‚0.5395124077074931 $loss/total_loss‚8.34419407783538 3loss/FBt2/assignment_loss‚0.0014895491788942414 .loss/FRt2/detection_loss‚1.215440116379859 0loss/SRqqt2/detection_loss‚0.607923800066481 .loss/FBt1/detection_loss‚0.933480075200681 0loss/FBt2/detection_loss‚0.46454552164492874 epoch‚16 % _timestamp‚1.7843460953495705e+09 2loss/SRqqt1/assignment_loss‚0.5087518804874978 1loss/SRqqt2/assignment_loss‚0.528197201993158 trainer/global_step‚6749 _runtime‚5754.042884942 _step‚555 1loss/SRqqt1/detection_loss‚0.9774692039594154í"7¿ À† 3loss/FBt1/assignment_loss‚8.539289535036687e-06 /loss/FRt1/detection_loss‚0.5395124077074931 $loss/total_loss‚8.34419407783538 3loss/FBt2/assignment_loss‚0.0014895491788942414 .loss/FRt2/detection_loss‚1.215440116379859 0loss/SRqqt2/detection_loss‚0.607923800066481 .loss/FBt1/detection_loss‚0.933480075200681 0loss/FBt2/detection_loss‚0.46454552164492874 epoch‚16 % _timestamp‚1.7843460953495705e+09 2loss/SRqqt1/assignment_loss‚0.5087518804874978 1loss/SRqqt2/assignment_loss‚0.528197201993158 trainer/global_step‚6749 _runtime‚5754.042884942 _step‚555 1loss/SRqqt1/detection_loss‚0.9774692039594154 /loss/FRt1/assignment_loss‚1.734549213457724 0loss/FRt2/assignment_loss‚0.8328265684697138«)&JU’Á‚(:‡ õëëҺɾ– gpu.0.gpu‚16.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºšŒÏ‚(:Ä õëëÒÚ““˜ disk.rbd1.out‚ 14.20703125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda2.in‚0 network.recv‚ 1.573538e+06 disk.drbd2165.out‚0 disk.dm-0.in‚11712.98046875) proc.memory.percent‚16.60630702972412 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0c0n1.out‚22181.798828125" disk.nvme0n1.in‚11712.98046875# proc.memory.rssMB‚ 2720.77734375$ memory_percent‚96.97871208190918 network.sent‚ 1.0631162e+07$ disk./.usageGB‚599.7803153991699 disk.sda3.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚11712.98046875* disk./.usagePercent‚40.251760215642726 disk.sda3.out‚ 184.0546875 disk.rbd0.out‚0 disk.sda1.in‚0! disk.dm-0.out‚22181.798828125 disk.rbd0.in‚0 disk.dm-1.out‚ 184.0546875' proc.memory.availableMB‚ 495.0078125 cpu‚26.15003689717225 disk.drbd2165.in‚0 disk.sda.out‚ 184.0546875$ disk.nvme0n1.out‚22181.798828125å[VÂ(:’ „ìëÒÓЖ• gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.496' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.89919999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SèÕÚÐÄ‚(:Å „ìëÒù視 disk.sda2.in‚0 disk.dm-1.in‚0 network.recv‚ 1.578532e+06 disk.sda3.out‚ 184.57421875 disk.rbd1.out‚ 14.26953125 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚22184.26171875$ disk./.usageGB‚599.7802429199219# proc.memory.rssMB‚ 2720.77734375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚11712.98046875) proc.memory.percent‚16.60630702972412 cpu‚26.150614206338897' proc.memory.availableMB‚ 494.8203125 disk.dm-1.out‚ 184.57421875 disk.sda.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚11712.98046875# disk.nvme0n1.out‚22184.26171875$ memory_percent‚96.97985649108887* disk./.usagePercent‚40.251755351499575 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚11712.98046875 disk.sda.out‚ 184.57421875 network.sent‚ 1.0681383e+07 disk.dm-0.out‚22184.26171875ŠL)Ô‘Å‚(:† “ìëÒ“ùÀ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öþd†ÓÆ‚(:È “ìëÒŸÛã• disk.sda2.in‚0& disk.nvme0n1.out‚22192.48193359375 disk.sda3.out‚ 184.8671875# disk.nvme0c0n1.in‚ 11713.9453125 disk.dm-0.in‚ 11713.9453125) proc.memory.percent‚16.60630702972412 network.recv‚ 1.581424e+06! disk.nvme0n1.in‚ 11713.9453125 disk.rbd1.out‚ 14.26953125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚26.151262035516275 network.sent‚ 1.0686045e+07 disk.sda1.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 184.8671875( disk.nvme0c0n1.out‚22192.48193359375 disk.sda.in‚0 disk.sda.out‚ 184.8671875$ disk./.usageGB‚599.7803153991699 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0# memory_percent‚96.9773530960083( proc.memory.availableMB‚ 495.23046875# proc.memory.rssMB‚ 2720.77734375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚22192.48193359375* disk./.usagePercent‚40.251760215642726o #2Ç- _wandbruntime‚5793  _runtime‚5793Ù‘³ ÉÈà ' lr-AdamW/pg1‚0.00034096832145075114 ' lr-AdamW/pg2‚0.00034096832145075114 trainer/global_step‚6799 % _timestamp‚1.7843461352517304e+09 _runtime‚5793.945389854 _step‚556U`“ÎÉÈ trainer/global_step‚6799 % _timestamp‚1.7843461352517304e+09 _runtime‚5793.945389854 _step‚556 ' lr-AdamW/pg1‚0.00034096832145075114 ' lr-AdamW/pg2‚0.00034096832145075114¬íÔoe2Ê- _wandbruntime‚5793  _runtime‚5793 œ QÌËîO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234948189840384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1032147908886528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53832118 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26520685 @:DeviceStatsMonitor.on_train_batch_s÷V9y($tart/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7167948463799 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1039326802388480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227694111852544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚557 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53832118 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53832116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1032147891847168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843461352528563e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62937441 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7254166564864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1026846463971328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039326802388480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234948297029632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039327160766976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039327160766976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7179251880448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7178910541312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034014395395767 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62927464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116769559 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62927464 trainer/global_step‚6799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1032147908886528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1032147891847168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53832116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26520419 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7168289013689 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039326802388480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234948297029632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039327160766976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039327160766976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7179251880448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7178910541312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034014395395767 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62927464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116769559 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62927464 trainer/global_step‚6799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1032147908886528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1032147891847168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53832116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26520419 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7168289013689 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚557 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53832118 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53832116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269­&&%2Í- _wandbruntime‚5794  _runtime‚5794»+ükþ&ÎøM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843461358649895e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26524488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1034161917121361 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚234982743891456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62946507 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1032295149916160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62936516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14610285 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7179923168256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14610542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7169300067523 _step‚558 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032295132876800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7179923168256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116776264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116776264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5794.558927332 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚6799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1032295132876800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26524228 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53839748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7180264514560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11913943 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227727559223296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026992974643200 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1032295149916160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62936516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14610285 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7179923168256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14610542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7169300067523 _step‚558 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032295132876800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7179923168256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116776264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116776264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5794.558927332 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚6799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚¨á¤w1032295132876800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26524228 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53839748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7180264514560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11913943 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227727559223296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026992974643200 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚234982743891456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0®ÙÖæ2Ð- _wandbruntime‚5794  _runtime‚5794§“õÑ‚ /loss/FRt2/assignment_loss‚0.875982557106944 /loss/FRt1/detection_loss‚0.5632656370941296 1loss/SRqqt1/detection_loss‚0.9611335863573266 %loss/total_loss‚8.032699945186064 /loss/FRt1/assignment_loss‚1.636867929548184 1loss/SRqqt2/detection_loss‚0.6273171758118449 /loss/FBt1/detection_loss‚0.4943251504849356 /loss/FBt2/detection_loss‚0.5449150708536618 epoch‚16 trainer/global_step‚6799 2loss/FBt2/assignment_loss‚0.011740583203488881 _step‚559 2loss/SRqqt1/assignment_loss‚0.5847001305598016 2loss/SRqqt2/assignment_loss‚0.5055521936308083 3loss/FBt1/assignment_loss‚0.0005375380707158902 .loss/FRt2/detection_loss‚1.226362392464223 % _timestamp‚1.7843461358698099e+09 _runtime‚5794.563124888öZÞ1 Ò‡ /loss/FRt1/detection_loss‚0.5632656370941296 1loss/SRqqt1/detection_loss‚0.9611335863573266 %loss/total_loss‚8.032699945186064 /loss/FRt1/assignment_loss‚1.636867929548184 1loss/SRqqt2/detection_loss‚0.6273171758118449 /loss/FBt1/detection_loss‚0.4943251504849356 /loss/FBt2/detection_loss‚0.5449150708536618 epoch‚16 trainer/global_step‚6799 2loss/FBt2/assignment_loss‚0.011740583203488881 _step‚559 2loss/SRqqt1/assignment_loss‚0.5847001305598016 2loss/SRqqt2/assignment_loss‚0.5055521936308083 3loss/FBt1/assignment_loss‚0.0005375380707158902 .loss/FRt2/detection_loss‚1.226362392464223 % _timestamp‚1.7843461358698099e+09 _runtime‚5794.563124888 /loss/FRt2/assignment_loss‚0.875982557106944¯¤É5 œÓ‚(:‘ ¢ìëÒ™Á¿• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.854' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.570800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߶à—ÓÔ‚(:È ¢ìëÒð·Â– disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚40.25176610381601( disk.nvme0c0n1.out‚22195.25244140625 disk.sda.in‚0 disk.sda3.out‚ 185.08203125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 185.08203125 disk.drbd2165.out‚0# disk.dm-0.out‚22195.25244140625 disk.sda.out‚ 185.08203125& disk.nvme0n1.out‚22195.25244140625# memory_percent‚97.0036268234253 cpu‚26.1576371724294 network.sent‚ 1.0703435e+07 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 11713.9453125 disk.sda2.in‚0( proc.memory.availableMB‚ 490.92578125 network.recv‚ 1.585027e+06! disk.nvme0n1.in‚ 11713.9453125 disk.sda3.in‚0* proc.memory.percent‚16.632485389709473# disk./.usageGB‚599.780403137207 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 14.33203125# proc.memory.rssMB‚ 2725.06640625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 11713.9453125¦sÐn2Õ- _wandbruntime‚5812  _runtime‚5812¡t1°=Öª{ 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20942054688930511 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041680268943309784 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9827876091003418 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02868536300957203 *jet/accuracy_2_of_4‚0.7973183393478394 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31808167695999146 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03182107210159302 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46336036920547485 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5684336423873901 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6836816072463989 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.09899009764194489 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10155564546585083 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7480967044830322 % _timestamp‚1.7843461538161013e+09 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2596603035926819 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4904003143310547 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27861154079437256 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6866547465324402 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6474980115890503 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2273644208908081 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06919905543327332 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6838642358779907 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996124804019928 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6883931159973145 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7035307884216309 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.560416579246521 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8563149571418762 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3257904350757599 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7203527688980103 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9030210375785828 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397034525871277 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892178773880005 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22601236402988434 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9970342516899109 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466977119445801 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7513375878334045 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9973735809326172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7814030647277832 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028108032420277596 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7758623957633972 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07352564483880997 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚16 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚6799 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537499666213989 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4070133864879608 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21375474333763123 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6800304651260376 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.023097663136882 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6394608616828918 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964032173156738 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6377513408660889 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3540854752063751 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895297288894653 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5696038007736206 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6603633165359497 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800293445587158 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999250054359436 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11989901959896088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5791082382202148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7471993565559387 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2258661389350891 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7575956583023071 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26281148195266724 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 _step‚560 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2143750935792923 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5056477189064026 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7844788432121277 +particle/specificity‚0.9116408824920654 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9920467138290405 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2292812168598175 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03086191788315773 )jet/accuracy_2_of_3‚0.586318850517273 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6651872396469116 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6397034525871277 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7730227112770081 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9944444298744202 /particle/accuracy_4_of_4‚0.9376680254936218 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9250779747962952 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6836816072463989 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7447020411491394 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7552719712257385 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020183298736810684 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.036601752042770386 +particle/sensitivity‚0.6334306001663208 particle/accuracy_2_of_5‚1 _runtime‚5812.510502786 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966562390327454 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913938641548157 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27992698550224304 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32195937633514404 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8732804656028748 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.215422123670578 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7557790279388428 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2668343484401703 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8534958958625793 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7238325476646423 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7043758034706116 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49476292729377747 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 (particle/accuracy‚0.8518919348716736 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7976424694061279 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5094103813171387 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08343871682882309 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02848978526890278 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159214973449707 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8520786762237549 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7validation_average_jet_accuracy‚0.39057785272598267 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20416520535945892 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27987512946128845 *jet/accuracy_1_of_4‚0.9801579713821411 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8703495264053345 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35279783606529236 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7143068909645081 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.221808522939682 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8563149571418762 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1011887639760971 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21326324343681335 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8839921355247498 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26887041330337524 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24289105832576752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8619426488876343 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4049021303653717 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7119231224060059 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7736771702766418 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6766027212142944 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19565565884113312 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 jet/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7616866827011108 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7113457322120667 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05340549349784851 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16149303317070007 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7965171337127686 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020183298736810684 *jet/accuracy_1_of_3‚0.9182509183883667 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26887041330337524 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05368489399552345 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8580310344696045 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7599838972091675 ("Purity/1FRt1SRqqt2FBt/event_purity‚0HÁ¿†²×¯{ =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7203527688980103 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9030210375785828 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397034525871277 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892178773880005 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22601236402988434 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9970342516899109 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466977119445801 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7513375878334045 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9973735809326172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7814030647277832 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028108032420277596 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7758623957633972 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07352564483880997 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚16 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚6799 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537499666213989 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4070133864879608 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21375474333763123 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6800304651260376 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.023097663136882 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6394608616828918 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964032173156738 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6377513408660889 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3540854752063751 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895297288894653 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5696038007736206 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6603633165359497 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4800293445587158 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999250054359436 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11989901959896088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5791082382202148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7471993565559387 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2258661389350891 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7575956583023071 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26281148195266724 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 _step‚560 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2143750935792923 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5056477189064026 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7844788432121277 +particle/specificity‚0.9116408824920654 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9920467138290405 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2292812168598175 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03086191788315773 )jet/accuracy_2_of_3‚0.586318850517273 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6651872396469116 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6397034525871277 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7730227112770081 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9944444298744202 /particle/accuracy_4_of_4‚0.9376680254936218 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9250779747962952 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6836816072463989 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7447020411491394 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7552719712257385 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020183298736810684 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.036601752042770386 +particle/sensitivity‚0.6334306001663208 particle/accuracy_2_of_5‚1 _runtime‚5812.510502786 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966562390327454 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913938641548157 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27992698550224304 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32195937633514404 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8732804656028748 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.215422123670578 =&Purity•X;"/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7557790279388428 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2668343484401703 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8534958958625793 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7238325476646423 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7043758034706116 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49476292729377747 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 (particle/accuracy‚0.8518919348716736 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7976424694061279 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5094103813171387 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08343871682882309 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02848978526890278 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159214973449707 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8520786762237549 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7validation_average_jet_accuracy‚0.39057785272598267 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20416520535945892 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27987512946128845 *jet/accuracy_1_of_4‚0.9801579713821411 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8703495264053345 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35279783606529236 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7143068909645081 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083134651184 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.221808522939682 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8563149571418762 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1011887639760971 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21326324343681335 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8839921355247498 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26887041330337524 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24289105832576752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8619426488876343 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4049021303653717 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7119231224060059 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7736771702766418 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6766027212142944 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19565565884113312 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 jet/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7616866827011108 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7113457322120667 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05340549349784851 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16149303317070007 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7965171337127686 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020183298736810684 *jet/accuracy_1_of_3‚0.9182509183883667 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26887041330337524 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05368489399552345 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8580310344696045 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7599838972091675 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20942054688930511 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041680268943309784 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9827876091003418 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02868536300957203 *jet/accuracy_2_of_4‚0.7973183393478394 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31808167695999146 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03182107210159302 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46336036920547485 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5684336423873901 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6836816072463989 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.09899009764194489 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10155564546585083 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7480967044830322 % _timestamp‚1.7843461538161013e+09 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2596603035926819 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4904003143310547 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27861154079437256 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6866547465324402 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 'particle/f_score‚0.6474980115890503 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2273644208908081 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06919905543327332 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6838642358779907 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996124804019928 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6883931159973145 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7035307884216309 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.560416579246521 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8563149571418762 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3257904350757599°#[¤ü“Ø‚(:ˆ ±ìëÒز” gpu.0.gpu‚95.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¯X.ÖÙ‚(:Ë ±ìëÒŽóª–! proc.memory.rssMB‚ 2725.265625) proc.memory.percent‚16.63370132446289 disk.rbd0.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚22448.59326171875) disk./.usagePercent‚40.25182216946601$ disk./.usageGB‚599.7812385559082 disk.sda3.out‚ 185.54296875 disk.drbd2165.out‚0 disk.sda.out‚ 185.54296875$ memory_percent‚95.47760486602783" disk.nvme0n1.in‚11814.03515625 disk.dm-1.in‚0 network.sent‚ 1.0753339e+07 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.16180817711828 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.33203125# disk.dm-0.out‚22448.59326171875( disk.nvme0c0n1.out‚22448.59326171875$ disk.nvme0c0n1.in‚11814.03515625 disk.dm-0.in‚11814.03515625 disk.sda1.out‚0 network.recv‚ 1.590385e+06 disk.sda3.in‚0 disk.dm-1.out‚ 185.54296875 disk.sda.in‚0( proc.memory.availableMB‚ 740.94921875ÅOžÚ‚(:“ ÀìëÒèÑž” gpu.0.gpu‚71.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Û.Æ·Û‚(:¬ ÀìëÒŽËÉ– disk.sda3.in‚0! proc.memory.rssMB‚ 2725.265625 disk.nvme0n1.in‚11815 cpu‚26.162202220147716 proc.cpu.threads‚17 disk.sda3.out‚ 185.88671875 disk.rbd1.out‚ 14.39453125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚22701.064453125 disk.sda.out‚ 185.88671875 disk.sda2.in‚0) proc.memory.percent‚16.63370132446289 disk.nvme0c0n1.in‚11815$ disk.nvme0n1.out‚22701.064453125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 185.88671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.0810673e+07 disk.sda1.in‚0( proc.memory.availableMB‚ 741.18359375* disk./.usagePercent‚40.251755863514646 disk.dm-0.in‚11815 disk.sda.in‚0 disk.sda2.out‚0$ disk./.usageGB‚599.7802505493164! disk.dm-0.out‚22701.064453125$ memory_percent‚95.47617435455322 network.recv‚ 1.595377e+06 disk.rbd0.out‚0MÊ(’Ü‚(:‡ ÏìëÒÌÚè“ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ#àû¶Ý‚(:« ÏìëÒ¡ˆ²–$ disk./.usageGB‚599.7786064147949 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 186.0390625 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme0n1.out‚22705.0068359375 disk.sda3.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚22705.0068359375 disk.sda.in‚0 disk.rbd1.out‚ 14.39453125 network.recv‚ 1.599075e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 186.0390625) proc.memory.percent‚16.63370132446289 disk.dm-0.in‚11815 disk.sda2.out‚0 disk.nvme0n1.in‚11815( proc.memory.availableMB‚ 741.17578125! proc.memory.rssMB‚ 2725.265625 cpu‚26.162853251680332 disk.sda1.in‚0 disk.dm-1.out‚ 186.0390625" disk.dm-0.out‚22705.0068359375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚11815$ memory_percent‚95.47622203826904 network.sent‚ 1.0815769e+07) disk./.usagePercent‚40.25164552426738“{ž2Þ- _wandbruntime‚5854  _runtime‚5854 w˜ÇßÁ & lr-AdamW/pg1‚0.0003394259094019006 & lr-AdamW/pg2‚0.0003394259094019006 trainer/global_step‚6849 % _timestamp‚1.7843461958397686e+09 _runtime‚5854.533359236 _step‚561M‚ôÌàÆ trainer/global_step‚6849 % _timestamp‚1.7843461958397686e+09 _runtime‚5854.533359236 _step‚561 & lr-AdamW/pg1‚0.0003394259094019006 & lr-AdamW/pg2‚0.0003394259094019006±°^GW2á- _wandbruntime‚5854  _runtime‚5854yŸ´wò'âìO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚6849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54334382 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7323417372160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048686849230336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26742909 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1048687207608832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236704879250944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1036080643739648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63452243 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5854.534585892 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54334384 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1041439527018496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1043317286004839 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7236625225831 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7247339251200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117796604 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229381480491008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12001968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7323495463424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041439509979136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14741194 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117786625 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54334384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63462220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63462220 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7247680590336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63452243 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1048687207608832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7236284675941 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7247339251200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236704975954432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54334382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14740941 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229381461878784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1036080660779008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚562 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12001971XGaG³#ãñO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236704879250944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1036080643739648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63452243 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5854.534585892 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54334384 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1041439527018496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1043317286004839 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7236625225831 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7247339251200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117796604 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229381480491008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12001968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7323495463424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041439509979136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14741194 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117786625 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54334384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63462220 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63462220 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7247680590336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63452243 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1048687207608832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7236284675941 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7247339251200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236704975954432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54334382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14740941 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229381461878784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1036080660779008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚562 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12001971 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚6849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54334382 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7323417372160 ZLDeviceSË¡èDtatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048686849230336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26742909 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1048687207608832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54342014 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7248351878144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7324443473920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117803309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461295 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14743251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7248351878144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54342016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚5855.148955943 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461295 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117813302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12003713 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1048835102886912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1036227154411520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7248693224448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236739437099008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1043464450141183 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54342014 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843461964552038e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041586751008768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54342016 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26746721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63471286 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229414909249536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63471286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7237636279665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚563 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048835461272576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1041586768048128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7237295729663 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169ŽÒÀ÷'æýM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14743251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7248351878144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54342016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚5855.148955943 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461295 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117813302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12003713 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813302 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1048835102886912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1036227154411520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7248693224448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236739437099008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1043464450141183 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54342014 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843461964552038e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041586751008768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54342016 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26746721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63471286 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229414909249536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63471286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7237636279665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚563 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048835461272576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1041586768048128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7237295729663 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54342014 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7248351878144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7324443473920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117803309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461295³ˆ»Ð2ç- _wandbruntime‚5855  _runtime‚5855:'Ú¿ è† /loss/FRt2/detection_loss‚1.2125492985791864 0loss/SRqqt1/detection_loss‚0.987071314153642 0loss/FRt1/assignment_loss‚1.6615392722547315 1loss/SRqqt2/detection_loss‚0.6105109632219583 /loss/FBt2/detection_loss‚0.3862384203564728 /loss/FRt2/assignment_loss‚1.030731245704081 4loss/FBt1/assignment_loss‚0.00039833881235157916 %loss/total_loss‚8.065629917175059 % _timestamp‚1.7843461964586513e+09 _step‚564 3loss/SRqqt1/assignment_loss‚0.46206008858832603 2loss/SRqqt2/assignment_loss‚0.5538461805578199 3loss/FBt2/assignment_loss‚0.0002046180379074209 /loss/FRt1/detection_loss‚0.5283948744002505 /loss/FBt1/detection_loss‚0.6320853025083326 epoch‚17 trainer/global_step‚6849 _runtime‚5855.152274869±bX°é‹ /loss/FRt2/assignment_loss‚1.030731245704081 4loss/FBt1/assignment_loss‚0.00039833881235157916 %loss/total_loss‚8.065629917175059 % _timestamp‚1.7843461964586513e+09 _step‚564 3loss/SRqqt1/assignment_loss‚0.46206008858832603 2loss/SRqqt2/assignment_loss‚0.5538461805578199 3loss/FBt2/assignment_loss‚0.0002046180379074209 /loss/FRt1/detection_loss‚0.5283948744002505 /loss/FBt1/detection_loss‚0.6320853025083326 epoch‚17 trainer/global_step‚6849 _runtime‚5855.152274869 /loss/FRt2/detection_loss‚1.2125492985791864 0loss/SRqqt1/detection_loss‚0.987071314153642 0loss/FRt1/assignment_loss‚1.6615392722547315 1loss/SRqqt2/detection_loss‚0.6105109632219583 /loss/FBt2/detection_loss‚0.3862384203564728´)–¿žê‚(:“ ÞìëÒæêÀ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åŽÿ²Öë‚(:Ë ÞìëҮ disk.rbd1.out‚ 14.45703125 network.sent‚ 1.0820702e+07 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚22712.20068359375 network.recv‚ 1.60237e+06 disk.sda.in‚0" disk.nvme0n1.in‚11815.83203125 disk.rbd0.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0& disk.nvme0n1.out‚22712.20068359375 disk.sda3.out‚ 186.26953125 disk.sda1.out‚0 disk.dm-1.out‚ 186.26953125 disk.drbd2165.in‚0 disk.sda.out‚ 186.26953125! proc.memory.rssMB‚ 2725.265625 disk.dm-0.in‚11815.83203125# disk.dm-0.out‚22712.20068359375$ disk.nvme0c0n1.in‚11815.83203125# disk./.usageGB‚599.780445098877 disk.rbd1.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 740.86328125* disk./.usagePercent‚40.251768919898886 disk.sda1.in‚0) proc.memory.percent‚16.63370132446289$ memory_percent‚95.47812938690186 cpu‚26.1632017780705721\’ì‚(:‡ íìëҴب” gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}‘%Ðí‚(:Å íìëÒÓéó– disk.rbd1.in‚0 disk.rbd0.in‚0# disk./.usageGB‚599.780330657959 disk.drbd2165.in‚0% disk.nvme0n1.out‚22720.6474609375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2725.265625 proc.cpu.threads‚17 disk.sda.out‚ 186.73046875) disk./.usagePercent‚40.25176123967286$ disk.nvme0c0n1.in‚11899.44921875 disk.rbd1.out‚ 14.45703125 disk.sda1.in‚0 network.sent‚ 1.0883599e+07 disk.sda2.in‚0" disk.nvme0n1.in‚11899.44921875 disk.sda3.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.63370132446289 network.recv‚ 1.607729e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 186.73046875 disk.dm-1.in‚0 disk.dm-1.out‚ 186.73046875" memory_percent‚95.479416847229' disk.nvme0c0n1.out‚22720.6474609375 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 740.65234375 disk.dm-0.in‚11899.44921875" disk.dm-0.out‚22720.6474609375 cpu‚26.16384092899672ßP;“î‚(:ˆ üìëÒó”å“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾šÿP×ï‚(:Ì üìëÒ›­î• disk.dm-1.in‚0 cpu‚26.164195241145855 network.sent‚ 1.0888601e+07 disk.sda2.out‚0 disk.drbd2165.in‚0# disk.dm-0.out‚22724.62548828125( proc.memory.availableMB‚ 494.41796875$ memory_percent‚96.98231220245361 disk.sda2.in‚0" disk.nvme0n1.in‚11899.44921875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 network.recv‚ 1.61116e+06$ disk./.usageGB‚599.7786979675293 disk.sda.in‚0 disk.sda3.out‚ 186.91015625 disk.rbd0.in‚0 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2725.265625* disk./.usagePercent‚40.251651668448204 disk.sda1.out‚0) proc.memory.percent‚16.63370132446289& disk.nvme0n1.out‚22724.62548828125 disk.dm-1.out‚ 186.91015625( disk.nvme0c0n1.out‚22724.62548828125 disk.rbd1.out‚ 14.51953125$ disk.nvme0c0n1.in‚11899.44921875 disk.dm-0.in‚11899.44921875 proc.cpu.threads‚17 disk.sda.out‚ 186.91015625Ïä~É2ð- _wandbruntime‚5895  _runtime‚5895!!`‹ÇñÁ trainer/global_step‚6899 % _timestamp‚1.7843462365507917e+09 _runtime‚5895.244368835 _step‚565 & lr-AdamW/pg1‚0.0003378761183475872 & lr-AdamW/pg2‚0.0003378761183475872[ªÌòÆ & lr-AdamW/pg1‚0.0003378761183475872 & lr-AdamW/pg2‚0.0003378761183475872 trainer/global_step‚6899 % _timestamp‚1.7843462365507917e+09 _runtime‚5895.244368835 _step‚565µ»“çƒ2ó- _wandbruntime‚5895  _runtime‚5895ô i…ôîO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54715984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63906695 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1048801561460736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5895.245556337 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7843462365518196e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54715984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1050693078496097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63916672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1048801578500096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14844394 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231052918417408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7298034753024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286901162849 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7298034753024 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1048801578500096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231052899805184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12089221 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1050693436085347 trainer/global_step‚6899 PCDeviceStatsMonitor.on_train_batch_start/allocation.smaç)Àoll_pool.freed‚63906695 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7298376092160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26933876 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118632656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056099596213760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238427640207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7374637407744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1048801561460736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26933612S/rù'õóO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118622677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7298376092160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14844655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7374721790464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7287241712739 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54715982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238427537212928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056099596213760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63916672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056099596213760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238427640207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7374637407744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1048801561460736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26933612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54715984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63906695 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1048801561460736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5895.245556337 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7843462365518196e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54715984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1050693078496097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63916672 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1048801578500096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14844394 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231052918417408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7298034753024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286901162849 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7298034753024 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1048801578500096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231052899805184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12089221 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1050693436085347 trainer/global_step‚6899 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63906695 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7298376092160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26933876 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118632656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649354 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54723614 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14846697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238462096453632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639361 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63925738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7299388726272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7299388726272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54723616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26937663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26937403 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚6899 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118639361 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54723616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54723614 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7299047379968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231086328563712 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1043552688005120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7299047379968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12090963 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14846440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238461993465856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231086347175936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7375664902144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1056247849870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050840600221691 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63925738 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048948819529728®eÐEøýM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639361 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63925738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7299388726272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7299388726272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54723616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26937663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26937403 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚6899 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118639361 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54723616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54723614 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7299047379968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231086328563712 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1043552688005120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7299047379968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12090963 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14846440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238461993465856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231086347175936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7375664902144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1056247849870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050840600221691 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63925738 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048948819529728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1048948802490368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649354 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54723614 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14846697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238462096453632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0·¦Áç2ù- _wandbruntime‚5895  _runtime‚5895»Ô¤Žú‚ /loss/FRt1/detection_loss‚0.5774056689217009 1loss/SRqqt2/detection_loss‚0.5881446841385995 trainer/global_step‚6899 .loss/FRt2/detection_loss‚1.324944754215426 _step‚568 %loss/total_loss‚8.119928239344313 epoch‚17 0loss/FRt1/assignment_loss‚1.6530052465358236 2loss/FBt1/assignment_loss‚0.011398079084877276 2loss/FBt2/assignment_loss‚0.013679906621984372 % _timestamp‚1.7843462371694107e+09 1loss/SRqqt1/detection_loss‚0.9377071899758529 _runtime‚5895.862718576 /loss/FBt1/detection_loss‚0.7039389403233695 /loss/FBt2/detection_loss‚0.5265351649443938 0loss/FRt2/assignment_loss‚0.7434111833572388 2loss/SRqqt1/assignment_loss‚0.5746640874905307 1loss/SRqqt2/assignment_loss‚0.465093333734516&² û‡ /loss/FBt1/detection_loss‚0.7039389403233695 /loss/FBt2/detection_loss‚0.5265351649443938 0loss/FRt2/assignment_loss‚0.7434111833572388 2loss/SRqqt1/assignment_loss‚0.5746640874905307 1loss/SRqqt2/assignment_loss‚0.465093333734516 /loss/FRt1/detection_loss‚0.5774056689217009 1loss/SRqqt2/detection_loss‚0.5881446841385995 trainer/global_step‚6899 .loss/FRt2/detection_loss‚1.324944754215426 _step‚568 %loss/total_loss‚8.119928239344313 epoch‚17 0loss/FRt1/assignment_loss‚1.6530052465358236 2loss/FBt1/assignment_loss‚0.011398079084877276 2loss/FBt2/assignment_loss‚0.013679906621984372 % _timestamp‚1.7843462371694107e+09 1loss/SRqqt1/detection_loss‚0.9377071899758529 _runtime‚5895.862718576¸}{d“ü‚(:ˆ ‹íëұʧ” gpu.0.gpu‚45.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïa#„Æý‚(:» ‹íëÒ͵°– disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 11900.28125$ memory_percent‚96.98255062103271 network.sent‚ 1.0893547e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 14.51953125 proc.cpu.threads‚17 disk.dm-1.out‚ 187.296875 disk.sda.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 11900.28125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 187.296875 disk.sda3.out‚ 187.296875( proc.memory.availableMB‚ 494.37890625 network.recv‚ 1.614731e+06* disk./.usagePercent‚40.251656788598886 cpu‚26.164786065216234 disk.dm-1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚22736.4775390625" disk.dm-0.out‚22736.4775390625 disk.nvme0n1.in‚ 11900.28125 disk.sda3.in‚0! proc.memory.rssMB‚ 2725.265625$ disk./.usageGB‚599.7787742614746 disk.rbd0.in‚0% disk.nvme0n1.out‚22736.4775390625) proc.memory.percent‚16.63370132446289Õr$“þ‚(:ˆ šíëÒÆõß” gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ž<šËÿ‚(:À šíëÒ›æÞ• disk.sda1.out‚0 disk.dm-1.out‚ 187.53515625" disk.dm-0.out‚22742.1904296875% disk.nvme0n1.out‚22742.1904296875 proc.cpu.threads‚17$ disk./.usageGB‚599.7791481018066 cpu‚26.165174619457776) disk./.usagePercent‚40.25168187733725 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 14.58203125) proc.memory.percent‚16.63370132446289 disk.sda.out‚ 187.53515625 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2725.265625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 187.53515625$ memory_percent‚96.98264598846436 network.recv‚ 1.619488e+06 disk.drbd2165.out‚0( proc.memory.availableMB‚ 494.36328125 network.sent‚ 1.0956621e+07 disk.sda.in‚0 disk.nvme0n1.in‚ 11900.28125 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 11900.28125' disk.nvme0c0n1.out‚22742.1904296875 disk.dm-0.in‚ 11900.28125 disk.rbd1.in‚0ýõ{2€- _wandbruntime‚5935  _runtime‚5935kÅÈ $ _timestamp‚1.784346276901773e+09 _runtime‚5935.595401522 _step‚569 ' lr-AdamW/pg1‚0.00033631904782799457 ' lr-AdamW/pg2‚0.00033631904782799457 trainer/global_step‚6949t¿ºzÍ‚Ç ' lr-AdamW/pg2‚0.00033631904782799457 trainer/global_step‚6949 $ _timestamp‚1.784346276901773e+09 _runtime‚5935.595401522 _step‚569 ' lr-AdamW/pg1‚0.00033631904782799457¹‰‚âU2ƒ- _wandbruntime‚5935  _runtime‚5935îDéô'„îO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1056163629981696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55097582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232724356343808 _runtime‚5935.596679003 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55097584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64371124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7349071593984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7425982167040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843462769029503e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058069228576605 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚570 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1056163612942336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12176468 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1050731710926848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7337858199647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12176471 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119458729 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7426066549760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1063512343197184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14947654 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7348730254848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64371124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64361147 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚6949 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55097582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64361147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1063512343197184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240150422893568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55097582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64361147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1063512343197184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240150422893568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1063512343197184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14947654 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7348730254848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64371124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ºeé¼2†- _wandbruntime‚5936  _runtime‚5936–Î.Ò¼‡öM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚6949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119485406 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7349742881792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7350084228096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14949685 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7427096748544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050878238638080 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64380190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240184881850880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232757785102336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1063660955239424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55105214 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1063660596853760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27128153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7427020761600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12178213 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7338869253481 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64370199 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050878221598720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7338528703479 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216750302199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55105214 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64380190 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1056310853971968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7350084228096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14949937 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056310871011328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27127898 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240184787251712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063660955239424 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚571 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058217107891561 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55105216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438ÓŒœ'ˆûM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55105214 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1063660596853760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27128153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7427020761600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12178213 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7338869253481 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64370199 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050878221598720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7338528703479 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216750302199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55105214 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64380190 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1056310853971968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7350084228096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14949937 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056310871011328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27127898 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240184787251712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063660955239424 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚571 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058217107891561 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55105216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚6949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119485406 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7349742881792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7350084228096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14949685 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7427096748544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050878238638080 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64380190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240184881850880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232757785102336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1063660955239424 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591»€«d2‰- _wandbruntime‚5936  _runtime‚5936Y° Š… %loss/total_loss‚8.220959497793856 2loss/SRqqt2/assignment_loss‚0.4865825576659195 3loss/FBt1/assignment_loss‚0.0023568585368815044 3loss/FBt2/assignment_loss‚0.0018422783606213756 /loss/FBt1/detection_loss‚0.7612975392370455 trainer/global_step‚6949 _runtime‚5936.214431473 2loss/SRqqt1/assignment_loss‚0.5246517033114935 0loss/SRqqt2/detection_loss‚0.608355045305317 0loss/FBt2/detection_loss‚0.43246181795130606 epoch‚17 _step‚572 0loss/FRt2/assignment_loss‚0.8539227857959993 /loss/FRt1/detection_loss‚0.6552918678963322 /loss/FRt2/detection_loss‚1.1904715027132735 1loss/SRqqt1/detection_loss‚1.0472441257593237 % _timestamp‚1.7843462775211496e+09 /loss/FRt1/assignment_loss‚1.656481415260344HÇ%‹Š %loss/total_loss‚8.220959497793856 2loss/SRqqt2/assignment_loss‚0.4865825576659195 3loss/FBt1/assignment_loss‚0.0023568585368815044 3loss/FBt2/assignment_loss‚0.0018422783606213756 /loss/FBt1/detection_loss‚0.7612975392370455 trainer/global_step‚6949 _runtime‚5936.214431473 2loss/SRqqt1/assignment_loss‚0.5246517033114935 0loss/SRqqt2/detection_loss‚0.608355045305317 0loss/FBt2/detection_loss‚0.43246181795130606 epoch‚17 _step‚572 0loss/FRt2/assignment_loss‚0.8539227857959993 /loss/FRt1/detection_loss‚0.6552918678963322 /loss/FRt2/detection_loss‚1.1904715027132735 1loss/SRqqt1/detection_loss‚1.0472441257593237 % _timestamp‚1.7843462775211496e+09 /loss/FRt1/assignment_loss‚1.656481415260344¼Ó®lÛ“Œ‚(:ˆ ©íëÒê´û“ gpu.0.gpu‚45.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rCZfÆ‚(:» ©íëÒàù©• disk.dm-0.out‚ 22755.1171875 disk.sda2.out‚0" disk.nvme0n1.out‚ 22755.1171875* disk./.usagePercent‚40.251692885661214$ disk.nvme0c0n1.out‚ 22755.1171875 disk.dm-1.in‚0 disk.dm-1.out‚ 187.81640625 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 187.81640625 disk.sda2.in‚0 disk.rbd1.out‚ 14.58203125! proc.memory.rssMB‚ 2725.265625 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 187.81640625 network.sent‚ 1.0961547e+07$ disk./.usageGB‚599.7793121337891 disk.sda1.in‚0 disk.nvme0n1.in‚ 11998.859375 disk.rbd1.in‚0( proc.memory.availableMB‚ 494.25390625) proc.memory.percent‚16.63370132446289" disk.nvme0c0n1.in‚ 11998.859375 disk.dm-0.in‚ 11998.859375 proc.cpu.threads‚17 network.recv‚ 1.623222e+06 disk.rbd0.out‚0$ memory_percent‚96.98331356048584 cpu‚26.165810410925765P¬„’’Ž‚(:‡ ¸íëһΓ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖvŸÑ‚(:Æ ¸íëÒ—Ží• disk.sda2.out‚0 disk.dm-1.out‚ 188.171875) proc.memory.percent‚16.63370132446289 proc.cpu.threads‚17 network.recv‚ 1.628471e+06" disk.nvme0n1.in‚11999.69140625 disk.rbd1.out‚ 14.64453125$ memory_percent‚96.98445796966553( proc.memory.availableMB‚ 494.06640625$ disk./.usageGB‚599.7791709899902 disk.dm-0.in‚11999.69140625# disk.dm-0.out‚22763.64892578125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚40.25168341338245 disk.sda3.out‚ 188.171875 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚11999.69140625( disk.nvme0c0n1.out‚22763.64892578125 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚22763.64892578125 cpu‚26.166225521828636 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 188.171875! proc.memory.rssMB‚ 2725.265625 network.sent‚ 1.1024572e+07 disk.rbd1.in‚0 disk.sda1.out‚0L³IÒœ‚(:‘ ÇíëÒߟæ“ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›FÐÉ‘‚(:¾ ÇíëÒàîÕ–( proc.memory.availableMB‚ 494.42578125 cpu‚26.16667761417199$ disk./.usageGB‚599.7792282104492 disk.drbd2165.out‚0 disk.dm-0.in‚11999.69140625 disk.rbd0.in‚0) proc.memory.percent‚16.63370132446289$ disk.nvme0c0n1.in‚11999.69140625" disk.dm-0.out‚22770.3779296875 network.recv‚ 1.632239e+06) disk./.usagePercent‚40.25168725349547 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚11999.69140625 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2725.265625 proc.cpu.threads‚17 disk.dm-1.out‚ 188.59375 disk.sda.out‚ 188.59375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.1029546e+07% disk.nvme0n1.out‚22770.3779296875' disk.nvme0c0n1.out‚22770.3779296875 disk.sda3.out‚ 188.59375 disk.rbd1.out‚ 14.64453125# memory_percent‚96.9822645187378(¬02’- _wandbruntime‚5976  _runtime‚5976¯¹³¯Ç“Á & lr-AdamW/pg1‚0.0003347547978508523 & lr-AdamW/pg2‚0.0003347547978508523 trainer/global_step‚6999 % _timestamp‚1.7843463173717506e+09 _runtime‚5976.065319641 _step‚5732m‘ûÌ”Æ trainer/global_step‚6999 % _timestamp‚1.7843463173717506e+09 _runtime‚5976.065319641 _step‚573 & lr-AdamW/pg1‚0.0003347547978508523 & lr-AdamW/pg2‚0.0003347547978508523½ÓmZ2•- _wandbruntime‚5976  _runtime‚5976éÖDBô'–îO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27314180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241873066404352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815599 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55479184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15050739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7399767095808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477381420544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚574 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1070925090180608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27314460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7477290746368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1065445378657113 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1058057261559808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7399425756672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120294781 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063525664423936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12263718 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚6999 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1063525664423936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7399425756672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241873175690752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063525681463296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7399767095808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55479182 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1065445736246363 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1058057244520448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388134136665 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64825576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234395794270208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55479182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12263721 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388474686555 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1070925448559104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120304760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55479184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5976.066597874 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063525681463296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234395775657984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1070925090180608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64825576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15050462 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063525681463296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234395775657984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1070925090180608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64825576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15050462 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063672922492928 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1071073343837184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7389145190387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15052745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1065592900382707 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64834642 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241907508292608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321458 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478303876096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚575 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5976.679896761 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55486814 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7400438383616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063672922492928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12265463 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1058203755192320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120321458 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55486816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234429223028736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15052475 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065593257972069 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27318211 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12265466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27317938 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784346317985885e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063672905453568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7400438383616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120311465 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64824651 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1058203772231680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55486816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64834642 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 Q®-Ç'šüM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27317938 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784346317985885e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063672905453568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7400438383616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120311465 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64824651 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1058203772231680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55486816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64834642 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063672922492928 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1071073343837184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7389145190387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15052745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1065592900382707 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64834642 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241907508292608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321458 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478303876096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚575 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚5976.679896761 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55486814 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7400438383616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063672922492928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12265463 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1058203755192320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120321458 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55486816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234429223028736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15052475 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065593257972069 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27318211 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12265466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664¿%î,2›- _wandbruntime‚5976  _runtime‚5976Ò¶Vã œƒ 3loss/FBt1/assignment_loss‚3.231458654132287e-05 1loss/SRqqt2/detection_loss‚0.5587116498566183 /loss/FBt1/detection_loss‚0.8404831242126724 _step‚576 2loss/SRqqt1/assignment_loss‚0.6230865909475222 2loss/SRqqt2/assignment_loss‚0.4973753688832621 3loss/FBt2/assignment_loss‚0.0026911223686620967 /loss/FRt1/detection_loss‚0.5978624470571309 _runtime‚5976.682011093 0loss/FRt2/assignment_loss‚0.8061740676017657 /loss/FRt2/detection_loss‚1.2078786350349502 /loss/FBt2/detection_loss‚0.5851549264909347 epoch‚17 % _timestamp‚1.7843463179886947e+09 /loss/FRt1/assignment_loss‚1.712488246126836 0loss/SRqqt1/detection_loss‚1.022611327580454 $loss/total_loss‚8.45454982074735 trainer/global_step‚6999vpöˆ 2loss/SRqqt2/assignment_loss‚0.4973753688832621 3loss/FBt2/assignment_loss‚0.0026911223686620967 /loss/FRt1/detection_loss‚0.5978624470571309 _runtime‚5976.682011093 0loss/FRt2/assignment_loss‚0.8061740676017657 /loss/FRt2/detection_loss‚1.2078786350349502 /loss/FBt2/detection_loss‚0.5851549264909347 epoch‚17 % _timestamp‚1.7843463179886947e+09 /loss/FRt1/assignment_loss‚1.712488246126836 0loss/SRqqt1/detection_loss‚1.022611327580454 $loss/total_loss‚8.45454982074735 trainer/global_step‚6999 3loss/FBt1/assignment_loss‚3.231458654132287e-05 1loss/SRqqt2/detection_loss‚0.5587116498566183 /loss/FBt1/detection_loss‚0.8404831242126724 _step‚576 2loss/SRqqt1/assignment_loss‚0.6230865909475222ÀÐtŸžž‚(:“ ÖíëÒÐß‚” gpu.0.gpu‚81.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ÝÊΟ‚(:à ÖíëÒÑï•( proc.memory.availableMB‚ 494.41796875 disk.rbd0.in‚0 network.sent‚ 1.1034484e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0) proc.memory.percent‚16.63370132446289 network.recv‚ 1.635761e+06 disk.sda.in‚0 disk.sda.out‚ 188.87109375 disk.sda3.out‚ 188.87109375 disk.rbd1.out‚ 14.76953125! proc.memory.rssMB‚ 2725.265625 disk.dm-1.out‚ 188.87109375$ memory_percent‚96.98231220245361 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 12000.65625( disk.nvme0c0n1.out‚22778.80908203125# disk.dm-0.out‚22778.80908203125 cpu‚26.167059011165883 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 12000.65625 disk.nvme0n1.in‚ 12000.65625& disk.nvme0n1.out‚22778.80908203125 proc.cpu.threads‚17) disk./.usagePercent‚40.25169058159341$ disk./.usageGB‚599.7792778015137 disk.rbd0.out‚0Szzž ‚(:“ åíëÒ§ô½“ gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.539' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.307799999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•–n%%¡‚(:» åíëÒ¶„‘– disk.sdaëJØ$¡3.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚22789.6982421875 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚26.167553238359044* disk./.usagePercent‚40.251692885661214$ disk./.usageGB‚599.7793121337891 disk.dm-1.out‚ 189.35546875 disk.drbd2165.in‚0 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2725.265625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 12095.125 disk.sda.out‚ 189.35546875$ memory_percent‚96.98007106781006) proc.memory.percent‚16.63370132446289 proc.cpu.threads‚17 network.sent‚ 1.1097605e+07 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 12095.125 disk.sda3.out‚ 189.35546875' disk.nvme0c0n1.out‚22789.6982421875 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚22789.6982421875 disk.rbd1.out‚ 14.76953125( proc.memory.availableMB‚ 494.78515625 network.recv‚ 1.641222e+06 disk.nvme0n1.in‚ 12095.125ìz‘‘¢‚(:† ôíëÒ¶¹Ó” gpu.0.gpu‚69.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07EùÇ£‚(:¼ ôíëÒÚ÷¡– disk.rbd0.out‚0& disk.nvme0n1.out‚22794.27685546875 network.recv‚ 1.644625e+06 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 12095.125 disk.sda.in‚0 disk.rbd1.in‚0 cpu‚26.168211703401223 proc.cpu.threads‚17( disk.nvme0c0n1.out‚22794.27685546875) proc.memory.percent‚16.63370132446289 disk.dm-1.out‚ 189.71484375 disk.sda1.in‚0# disk.dm-0.out‚22794.27685546875 disk.nvme0n1.in‚ 12095.125 disk.sda3.in‚0$ memory_percent‚96.98171615600586! proc.memory.rssMB‚ 2725.265625& proc.memory.availableMB‚ 494.515625 disk.dm-0.in‚ 12095.125 disk.sda.out‚ 189.71484375 disk.sda2.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 189.71484375 disk.drbd2165.out‚0 network.sent‚ 1.1102599e+07* disk./.usagePercent‚40.251666516885194$ disk./.usageGB‚599.7789192199707 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 14.83203125 disk.dm-1.in‚0Úm>€2¤- _wandbruntime‚6016  _runtime‚6016æs­†É¥Ã ' lr-AdamW/pg1‚0.00033318346888501363 ' lr-AdamW/pg2‚0.00033318346888501363 trainer/global_step‚7049 % _timestamp‚1.7843463579607613e+09 _runtime‚6016.654349155 _step‚577’£èÎ¦È ' lr-AdamW/pg1‚0.00033318346888501363 ' lr-AdamW/pg2‚0.00033318346888501363 trainer/global_step‚7049 % _timestamp‚1.7843463579607613e+09 _runtime‚6016.654349155 _step‚577Á®§Ê2§- _wandbruntime‚6016  _runtime‚6016ÍæBò'¨ìO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1070887715905536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1065382795153408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12350968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1072821886326871 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65280028 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚578 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243595914853376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070887732944896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1078337837164032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121140812 # _timestamp‚1.78434635796186e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12350971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1070887715905536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1065382778114048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55860782 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55860784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236067232196608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7438750623573 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65270051 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7450462597632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1078338195542528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1078338195542528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7450121258496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55860782 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176*ïc÷'©ñO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55860782 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1070887715905536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1065382795153408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12350968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1072821886326871 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65280028 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚578 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243595914853376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070887732944896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1078337837164032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121140812 # _timestamp‚1.78434635796186e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12350971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1070887715905536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1065382778114048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55860782 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55860784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236067232196608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7438750623573 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65270051 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7450462597632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1078338195542528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1078338195542528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7450121258496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576Âü勦2ª- _wandbruntime‚6017  _runtime‚60171g•°¤$«øM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1071034956935168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55868414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12352716 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7529707553280 trainer/global_step‚7049 _runtime‚6017.269406409 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1072969408052577 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121147517 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7451475231744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27508858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12352713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15155889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65289094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072969050463215 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 % _timestamp‚1.7843463585755684e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157510 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243630368508416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65279103 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55868414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236100660955136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1071034956935168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15156142 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚579 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1078486449206272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1065529305825280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7451133885440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071034973974528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65279103 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236100642342912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27508602 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1078486449206272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065529288785920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157510 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243630368508416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65279103 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55868414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236100660955136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1071034956935168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15156142 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚579 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1078486449206272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1065529305825280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7451133885440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071034973974528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65279103 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236100642342912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27508602 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1078486449206272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065529288785920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7451475231744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27508858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12352713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15155889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65289094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072969050463215 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169Ã$á¼2­- _wandbruntime‚6017  _runtime‚6017]Þ ®ƒ 0loss/FRt2/assignment_loss‚0.8448301219358677 trainer/global_step‚7049 2loss/FBt2/assignment_loss‚0.004376320464515716 .loss/FBt1/detection_loss‚0.582485556602478 &loss/total_loss‚7.9924529338357235 epoch‚17 $ _timestamp‚1.784346358578761e+09 0loss/FRt1/assignment_loss‚1.6212121630963097 2loss/SRqqt2/assignment_loss‚0.4375918761038406 2loss/FBt1/assignment_loss‚0.019264585063817005 /loss/FRt1/detection_loss‚0.5149914801315456 1loss/SRqqt2/detection_loss‚0.5634812978083208 _runtime‚6017.272058586 _step‚580 2loss/SRqqt1/assignment_loss‚0.5521152722437385 .loss/FRt2/detection_loss‚1.329658634792417 1loss/SRqqt1/detection_loss‚1.0831611868498832 0loss/FBt2/detection_loss‚0.43928443874298995åk ½¯ˆ 2loss/FBt1/assignment_loss‚0.019264585063817005 /loss/FRt1/detection_loss‚0.5149914801315456 1loss/SRqqt2/detection_loss‚0.5634812978083208 _runtime‚6017.272058586 _step‚580 2loss/SRqqt1/assignment_loss‚0.5521152722437385 .loss/FRt2/detection_loss‚1.329658634792417 1loss/SRqqt1/detection_loss‚1.0831611868498832 0loss/FBt2/detection_loss‚0.43928443874298995 0loss/FRt2/assignment_loss‚0.8448301219358677 trainer/global_step‚7049 2loss/FBt2/assignment_loss‚0.004376320464515716 .loss/FBt1/detection_loss‚0.582485556602478 &loss/total_loss‚7.9924529338357235 epoch‚17 $ _timestamp‚1.784346358578761e+09 0loss/FRt1/assignment_loss‚1.6212121630963097 2loss/SRqqt2/assignment_loss‚0.4375918761038406Ä=¹Ë“°‚(:ˆ ƒîëÒŒ®Ž• gpu.0.gpu‚71.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 MV`Ò±‚(:Ç ƒîëÒÖõ¬– disk.dm-1.out‚ 190.1953125$ memory_percent‚96.98185920715332* disk./.usagePercent‚40.251666772892726 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 190.1953125 disk.sda2.in‚0 network.sent‚ 1.1120684e+07# disk./.usageGB‚599.778923034668& disk.nvme0n1.out‚22806.11279296875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2725.265625 disk.drbd2165.in‚0 disk.dm-0.in‚12096.08984375# disk.dm-0.out‚22806.11279296875 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚26.168661025483097 disk.sda1.out‚0 network.recv‚ 1.64822e+06( disk.nvme0c0n1.out‚22806.11279296875 disk.sda3.out‚ 190.1953125$ disk.nvme0c0n1.in‚12096.08984375 disk.dm-1.in‚0) proc.memory.percent‚16.63370132446289" disk.nvme0n1.in‚12096.08984375 disk.rbd1.out‚ 14.83203125 disk.rbd0.in‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 494.4921875v×TGž²‚(:“ ’îëÒĵҔ gpu.0.gpu‚61.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕŠXг‚(:Å ’îëÒ©ÇÕ disk.dm-0.in‚ 12096.921875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 494.23046875& disk.nvme0n1.out‚22814.99267578125 disk.sda3.in‚0 disk.rbd1.out‚ 14.89453125 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.out‚ 190.46484375 disk.nvme0n1.in‚ 12096.921875 disk.dm-1.in‚0 disk.dm-1.out‚ 190.46484375 network.sent‚ 1.1170662e+07$ disk./.usageGB‚599.7789192199707" disk.nvme0c0n1.in‚ 12096.921875( disk.nvme0c0n1.out‚22814.99267578125 disk.sda.in‚0 disk.rbd1.in‚0# memory_percent‚96.9834566116333! proc.memory.rssMB‚ 2725.265625* disk./.usagePercent‚40.251666516885194# disk.dm-0.out‚22814.99267578125 disk.sda3.out‚ 190.46484375 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.63370132446289 network.recv‚ 1.653313e+06 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚26.16923973004565pn2´- _wandbruntime‚6056  _runtime‚6056³_ƒ¤ÉµÃ ' lr-AdamW/pg1‚0.00033160516185400167 ' lr-AdamW/pg2‚0.00033160516185400167 trainer/global_step‚7099 % _timestamp‚1.7843463982767303e+09 _runtime‚6056.970294258 _step‚581‚pfkÎ¶È % _timestamp‚1.7843463982767303e+09 _runtime‚6056.970294258 _step‚581 ' lr-AdamW/pg1‚0.00033160516185400167 ' lr-AdamW/pg2‚0.00033160516185400167 trainer/global_step‚7099Ål$2·- _wandbruntime‚6056  _runtime‚6056?½¼Ïô'¸îO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1078249784426496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1085750942525952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7500816760320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7843463982777278e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56242384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078249767387136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7500816760320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65734480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1085750584147456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1072708311707648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7489367110481 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7501158099456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65724503 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65734480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27695134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15257174 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6056.971461942 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27695395 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080197678818129 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078249784426496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1085750942525952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7579703151104 trainer/global_step‚7099 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7579787533824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12438221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237738670123008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080198036407379 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1078249767387136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7501158099456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237738651510784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7489707660371 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65724503 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072708328747008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56242384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258a¥õð/¹óO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65724503 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65734480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27695134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15257174 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6056.971461942 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27695395 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080197678818129 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078249784426496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1085750942525952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7579703151104 trainer/global_step‚7099 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7579787533824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12438221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237738670123008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080198036407379 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1078249767387136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7501158099456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237738651510784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7489707660371 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65724503 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocÿ$–ÉÊ ated‚1072708328747008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56242384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1078249784426496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1085750942525952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7500816760320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7843463982777278e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56242384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078249767387136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7500816760320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65734480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1085750584147456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1072708311707648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7489367110481 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7501158099456Æ>ø¡H2º- _wandbruntime‚6057  _runtime‚6057L³®ÿ&»ùM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27698926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7580710016000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27699210 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245352910050304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1080345200543723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12439963 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56250014 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65733555 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1078397008416768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121993562 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237772098881536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7490378164203 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1078397025456128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993562 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237772080269312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65743546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7502170733568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7501829387264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1085899196189696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12439966 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1078397025456128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501829387264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7580811168768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7099 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65743546P©eÃ'¼þM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1078397008416768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121993562 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237772098881536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7490378164203 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1078397025456128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993562 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237772080269312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65743546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7502170733568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7501829387264 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1085899196189696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12439966 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1078397025456128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501829387264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7580811168768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7099 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65743546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27698926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7580710016000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27699210 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245352910050304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1080345200543723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12439963 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56250014 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65733555 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÇÅ9ËR2½- _wandbruntime‚6057  _runtime‚6057†tòe ¾„ /loss/FRt1/assignment_loss‚1.690528447782644 /loss/FRt1/detection_loss‚0.5532082445011822 /loss/FRt2/detection_loss‚1.1764507511196125 2loss/FBt2/assignment_loss‚0.023368362745979237 1loss/SRqqt1/detection_loss‚0.9753503620127203 0loss/FBt2/detection_loss‚0.45519516220681705 trainer/global_step‚7099 _runtime‚ 6057.58708268 _step‚584 2loss/SRqqt1/assignment_loss‚0.5155830240712306 2loss/SRqqt2/assignment_loss‚0.4736683787881968 2loss/FBt1/assignment_loss‚0.023219425619223747 0loss/FBt1/detection_loss‚0.45072329217611357 %loss/total_loss‚7.825135404560703 0loss/FRt2/assignment_loss‚0.8528900785185748 1loss/SRqqt2/detection_loss‚0.6349498750184074 epoch‚17 % _timestamp‚1.7843463988937778e+09©íÈÝ¿‰ 2loss/FBt1/assignment_loss‚0.023219425619223747 0loss/FBt1/detection_loss‚0.45072329217611357 %loss/total_loss‚7.825135404560703 0loss/FRt2/assignment_loss‚0.8528900785185748 1loss/SRqqt2/detection_loss‚0.6349498750184074 epoch‚17 % _timestamp‚1.7843463988937778e+09 /loss/FRt1/assignment_loss‚1.690528447782644 /loss/FRt1/detection_loss‚0.5532082445011822 /loss/FRt2/detection_loss‚1.1764507511196125 2loss/FBt2/assignment_loss‚0.023368362745979237 1loss/SRqqt1/detection_loss‚0.9753503620127203 0loss/FBt2/detection_loss‚0.45519516220681705 trainer/global_step‚7099 _runtime‚ 6057.58708268 _step‚584 2loss/SRqqt1/assignment_loss‚0.5155830240712306 2loss/SRqqt2/assignment_loss‚0.4736683787881968ÈÁ5žÀ‚(:“ ¡îëÒ“ÐÜ” gpu.0.gpu‚86.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.772600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rDîÅÁ‚(:º ¡îëÒºÕ•—# disk.nvme0n1.out‚22820.62890625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 190.73046875 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚22820.62890625' proc.memory.availableMB‚ 494.4296875 disk.dm-0.in‚ 12183.90625 disk.sda2.out‚0 disk.rbd1.out‚ 14.89453125$ memory_percent‚96.98224067687988 network.sent‚ 1.1175542e+07$ disk./.usageGB‚599.7793121337891 disk.nvme0n1.in‚ 12183.90625 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 12183.90625 disk.dm-0.out‚22820.62890625 disk.sda.out‚ 190.73046875 disk.sda2.in‚0 disk.sda3.out‚ 190.73046875 disk.rbd0.out‚0 network.recv‚ 1.656998e+06! proc.memory.rssMB‚ 2725.265625) proc.memory.percent‚16.63370132446289 cpu‚26.169629331268407 proc.cpu.threads‚17* disk./.usagePercent‚40.251692885661214 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0ä0rý“‚(:ˆ °îëÒ™ïÉ“ gpu.0.gpu‚35.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rjÐÂ(:Å °îëÒ×”þ–) proc.memory.percent‚16.63370132446289$ disk./.usageGB‚599.7793083190918 disk.rbd1.out‚ 14.95703125 disk.dm-1.in‚0 network.sent‚ 1.1238477e+07 disk.rbd1.in‚0 network.recv‚ 1.662031e+06 disk.sda1.out‚0 disk.sda.in‚0& disk.nvme0n1.out‚22828.49072265625 proc.cpu.threads‚17 disk.dm-1.out‚ 191.046875 disk.sda2.in‚0( proc.memory.availableMB‚ 494.49609375) disk./.usagePercent‚40.25169262965368 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚12184.87109375" disk.nvme0n1.in‚12184.87109375 cpu‚26.17019716489961 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚12184.87109375! proc.memory.rssMB‚ 2725.265625 disk.rbd0.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚22828.49072265625# disk.dm-0.out‚22828.49072265625 disk.sda.out‚ 191.046875 disk.sda3.out‚ 191.046875 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚96.98183536529541xŧ֓Ä‚(:ˆ ¿îëÒßåò” gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ê—jÇÅ‚(:¼ ¿îëÒÚ⸖ disk.sda1.out‚0 disk.nvme0n1.in‚ 12185.703125 network.sent‚ 1.1243524e+07 disk.sda1.in‚0( disk.nvme0c0n1.out‚22837.09423828125 disk.sda2.out‚0 network.recv‚ 1.66558e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 12185.703125$ memory_percent‚96.98371887207031) proc.memory.percent‚16.63370132446289 proc.cpu.threads‚17& disk.nvme0n1.out‚22837.09423828125 disk.sda3.out‚ 191.390625 disk.drbd2165.out‚0 disk.sda.out‚ 191.390625 disk.sda2.in‚0! proc.memory.rssMB‚ 2725.265625$ disk./.usageGB‚599.7795333862305 disk.rbd1.in‚0 disk.rbd1.out‚ 14.95703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 191.390625 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚40.251707734098204# disk.dm-0.out‚22837.09423828125" disk.nvme0c0n1.in‚ 12185.703125$ proc.memory.availableMB‚494.1875 cpu‚26.170601773437443ŸxÖ_2Æ- _wandbruntime‚6097  _runtime‚6097QÅ]ÇÇÁ % _timestamp‚1.7843464387717986e+09 _runtime‚6097.465422039 _step‚585 & lr-AdamW/pg1‚0.0003300199781295277 & lr-AdamW/pg2‚0.0003300199781295277 trainer/global_step‚7149Ò–þ¦ÌÈÆ trainer/global_step‚7149 % _timestamp‚1.7843464387717986e+09 _runtime‚6097.465422039 _step‚585 & lr-AdamW/pg1‚0.0003300199781295277 & lr-AdamW/pg2‚0.0003300199781295277ÉzÔ[ä2É- _wandbruntime‚6097  _runtime‚6097Ò7ýb! ÊîO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7551512262144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122802937 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812916 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1087574186487887 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 % _timestamp‚1.7843464387729642e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239410108049408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087573828898637 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7539983597389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12525468 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1093163689509376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMn“š~Óonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12525471 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56623982 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7551512262144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239410089437184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56623984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1093163331130880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7630879456768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66178955 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56623984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66188932 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080033862340608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1093163689509376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7551853601280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085611835908096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27885629 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085611835908096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7630972228096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080033845301248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6097.466698889 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27885894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1093163331130880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7551853601280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247040968893952 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7540324147279 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085611818868736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66188932 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247041080277504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15360161¾¹ç\ù'ËóO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085611818868736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66188932 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247041080277504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15360161 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7551512262144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122802937 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812916 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1087574186487887 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 % _timestamp‚1.7843464387729642e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239410108049408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087573828898637 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7539983597389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12525468 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1093163689509376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12525471 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56623982 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7551512262144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239410089437184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56623984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1093163331130880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7630879456768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66178955 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56623984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66188932 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080033862340608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1093163689509376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7551853601280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085611835908096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27885629 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085611835908096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7630972228096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080033845301248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6097.466698889 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27885894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1093163331130880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7551853601280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247040968893952 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7540324147279ÊV Ð×2Ì- _wandbruntime‚6098  _runtime‚6098•Oû&ÍõM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15362463 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247075527573504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239443536807936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7541335201113 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56631614 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66188007 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7631990765568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122819621 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1093311943173120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085759059898368 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087721708213593 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085759076937728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56631614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085759076937728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7552524889088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239443518195712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27889409 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247075422488576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56631616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1093311584787456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12527216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829614 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66197998 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1085759059898368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56631616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66197998 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7552866235392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122829614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122819621 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27889679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085759076937728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7552524889088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239443518195712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27889409 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247075422488576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56631616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1093311584787456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12527216 XJDeviceStatsMonitor.on_train_batch_end/inactive_splittK¾_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829614 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66197998 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1085759059898368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56631616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66197998 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7552866235392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122829614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122819621 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27889679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7541335201113 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56631614 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66188007 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7631990765568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122819621 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1093311943173120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085759059898368 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087721708213593 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085759076937728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56631614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704Ë"WI2Ï- _wandbruntime‚6098  _runtime‚6098,pvÕ Ð„ 0loss/FRt1/assignment_loss‚1.6193039443264616 /loss/FRt2/detection_loss‚1.2175629250097757 1loss/SRqqt2/detection_loss‚0.5980135479867023 trainer/global_step‚7149 $ _timestamp‚1.784346439389834e+09 3loss/SRqqt2/assignment_loss‚0.49009017907025876 3loss/FBt1/assignment_loss‚0.0008504357428864573 3loss/FBt2/assignment_loss‚6.319040515430469e-05 %loss/total_loss‚7.986301894896736 /loss/FRt1/detection_loss‚0.6102220375575386 /loss/FBt1/detection_loss‚0.7138879001140594 /loss/FBt2/detection_loss‚0.3679604667327206 epoch‚17 0loss/FRt2/assignment_loss‚0.9074677765842468 3loss/SRqqt1/assignment_loss‚0.49154805290791814 0loss/SRqqt1/detection_loss‚0.969331438459014 _runtime‚ 6098.0831155 _step‚588²¤vщ /loss/FRt1/detection_loss‚0.6102220375575386 /loss/FBt1/detection_loss‚0.7138879001140594 /loss/FBt2/detection_loss‚0.3679604667327206 epoch‚17 0loss/FRt2/assignment_loss‚0.9074677765842468 3loss/SRqqt1/assignment_loss‚0.49154805290791814 0loss/SRqqt1/detection_loss‚0.969331438459014 _runtime‚ 6098.0831155 _step‚588 0loss/FRt1/assignment_loss‚1.6193039443264616 /loss/FRt2/detection_loss‚1.2175629250097757 1loss/SRqqt2/detection_loss‚0.5980135479867023 trainer/global_step‚7149 $ _timestamp‚1.784346439389834e+09 3loss/SRqqt2/assignment_loss‚0.49009017907025876 3loss/FBt1/assignment_loss‚0.0008504357428864573 3loss/FBt2/assignment_loss‚6.319040515430469e-05 %loss/total_loss‚7.986301894896736ÌpDHåœÒ‚(:‘ ÎîëÒçÅ” gpu.0.gpu‚33.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûØÈÅÓ‚(:º ÎîëÒÛÖ–% disk.nvme0n1.out‚22839.9775390625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚22839.9775390625 disk.sda1.in‚0) proc.memory.percent‚16.63370132446289 network.recv‚ 1.669015e+06$ disk./.usageGB‚599.7794570922852 disk.nvme0n1.in‚ 12185.703125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 cpu‚26.17108508222037 network.sent‚ 1.1248401e+07$ memory_percent‚96.98381423950195& proc.memory.availableMB‚ 494.171875 disk.rbd0.out‚0 disk.dm-1.out‚ 191.984375 disk.sda.out‚ 191.984375) disk./.usagePercent‚40.25170261394752 disk.rbd1.out‚ 15.01953125" disk.nvme0c0n1.in‚ 12185.703125" disk.dm-0.out‚22839.9775390625! proc.memory.rssMB‚ 2725.265625 disk.sda2.out‚0 disk.sda3.out‚ 191.984375 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 12185.703125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0(-3TÔ‚(:’ ÝîëÒ¤ñæ“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|wŠÊÕ‚(:¿ ÝîëÒÍ€§– disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚22848.650390625& proc.memory.availableMB‚ 494.171875 network.sent‚ 1.1311312e+07 disk.rbd1.out‚ 15.01953125 disk.sda2.in‚0" disk.nvme0n1.in‚12224.66796875! proc.memory.rssMB‚ 2725.265625) proc.memory.percent‚16.63370132446289 disk.sda1.out‚0 disk.sda3.out‚ 192.30859375 cpu‚26.17151896237195 disk.drbd2165.out‚0 disk.dm-0.in‚ 12224.828125! disk.dm-0.out‚22848.650390625 disk.sda.in‚0 disk.sda3.in‚0) disk./.usagePercent‚40.25171567033177 disk.sda2.out‚0 network.recv‚ 1.674077e+06" disk.nvme0c0n1.in‚ 12224.390625$ memory_percent‚96.98381423950195 proc.cpu.threads‚17 disk.dm-1.out‚ 192.30859375$ disk./.usageGB‚599.7796516418457 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚22848.650390625 disk.sda.out‚ 192.308593752aè«“Ö‚(:ˆ ìîëÒÚÏ´” gpu.0.gpu‚62.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚215.228' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.0456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01TW^Ìׂ(:Á ìîëÒñéÒ•! proc.memory.rssMB‚ 2725.265625 disk.sda2.in‚0" disk.nvme0n1.in‚12245.89453125" disk.dm-0.out‚22856.2431640625 disk.sda.out‚ 192.609375 cpu‚26.17204030511905 proc.cpu.threads‚17 network.sent‚ 1.1316455e+07 disk.sda1.in‚0$ disk.nvme0c0n1.in‚12245.89453125 disk.dm-0.in‚12245.89453125) proc.memory.percent‚16.63370132446289 disk.sda.in‚0$ memory_percent‚96.98383808135986 disk.rbd1.out‚ 15.14453125) disk./.usagePercent‚40.25170466200779$ disk./.usageGB‚599.7794876098633 disk.sda2.out‚0 disk.sda3.out‚ 192.609375 disk.dm-1.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚22856.2431640625 network.recv‚ 1.67739e+06% disk.nvme0n1.out‚22856.2431640625 disk.sda3.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 494.16796875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 192.609375bÉÞ2Ø- _wandbruntime‚6137  _runtime‚6137£ŒTÅÙ¿ & lr-AdamW/pg2‚0.0003284280195249802 trainer/global_step‚7199 # _timestamp‚1.78434647910069e+09 _runtime‚6137.794281731 _step‚589 & lr-AdamW/pg1‚0.0003284280195249802òӶÊÚÄ trainer/global_step‚7199 # _timestamp‚1.78434647910069e+09 _runtime‚6137.794281731 _step‚589 & lr-AdamW/pg1‚0.0003284280195249802 & lr-AdamW/pg2‚0.0003284280195249802;”2Û- _wandbruntime‚6137  _runtime‚6137u´Y]ô'ÜîO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66643384 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1100576436492800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248763701587456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57005582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7602207763968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1100576436492800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57005584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1100576078114304 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1094949978979145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66643384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28076676 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12612718 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7602549103104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7602549103104 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1087359378894848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚7199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7590940634187 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1092973887389696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248763804582400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12612721 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1100576078114304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241081545975808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0/ˆÚù'ÝóO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28076934 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66633407 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1092973870350336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682174223872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123638989 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7590940634187 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1092973887389696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248763804582400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12612721 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1100576078114304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241081545975808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66643384 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1100576436492800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248763701587456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57005582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7602207763968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1100576436492800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57005584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1100576078114304 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1094949978979145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66643384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28076676 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12612718 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7602549103104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7602549103104 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1087359378894848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚7199 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66642459 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591611138019 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_ba¶¦¬"tch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1100724690156544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095097500704739 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843464797144492e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12614463 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241114974734336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57013214 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66652450 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7603561737216 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093121111379968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15465989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7683197343744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚6138.408208675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57013216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1095097858294101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248798153465856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7683281719296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12614466 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57013214 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66652450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28080452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66642459 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591611138019 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1100724690156544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095097500704739 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843464797144492e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12614463 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241114974734336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57013214 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66652450 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7603561737216 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093121111379968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15465989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7683197343744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚6138.408208675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57013216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1095097858294101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248798153465856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7683281719296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12614466 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57013214 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66652450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28080452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_1_of_2‚0.7555528879165649 particle/accuracy_1_of_2‚1 *loss/val_total_loss‚7.9272891336445275 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36853286623954773 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6291236281394958 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22720254957675934 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5511418581008911 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.778626561164856 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6578007936477661 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237908005714417 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036031972616910934 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7795096635818481 +jet/accuracy_3_of_4‚0.08533445745706558 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.313079297542572 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2790018618106842 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7205245494842529 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8677479028701782 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8677479028701782 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_4_of_4‚0.9486338496208191 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7292895317077637 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025794360786676407 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6968379616737366 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11097162216901779 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.34395334124565125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6996828317642212 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7852985858917236 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28881460428237915 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_5_of_5‚0 _step‚593 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948212504386902 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848228693008423 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28881460428237915 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6780388951301575 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7804210186004639 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 )jet/accuracy_2_of_2‚0.155134379863739 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7847316265106201 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.611487865447998 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2135114222764969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954733848571777 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862183332443237 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29751646518707275 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2159487009048462 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45816153287887573 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038506485521793365 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37128788232803345 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6177191138267517 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5115148425102234 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06109023094177246 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4838956296443939 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6997324824333191 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25265732407569885 +jet/accuracy_3_of_3‚0.07130808383226395 _runtime‚6156.549218161 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48875975608825684 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9931118488311768 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5675630569458008 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07374705374240875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6234888434410095 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2966991364955902 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2319529801607132 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.313265860080719 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.658383846282959 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 (particle/accuracy‚0.8442537188529968 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.968169093132019 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6234888434410095 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7465888261795044 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04772433638572693 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19670291244983673 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19704081118106842 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5675312876701355 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4966926574707031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2990580201148987 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 jet/accuracy_4_of_5‚0 % _timestamp‚1.7843464978549354e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2432282567024231 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28588372468948364 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633 MÂ3949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6391420364379883 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22292514145374298 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942053556442261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840845465660095 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.378718763589859 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7104777693748474 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.563442587852478 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22569754719734192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7985072135925293 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7748858332633972 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2703286409378052 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_4‚0.9720036387443542 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6793110370635986 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2172321379184723 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6693258285522461 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9124894738197327 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 .particle/accuracy_3_of_4‚0.993424117565155 *particle/sensitivity‚0.623106837272644 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6374256610870361 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5701592564582825 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02600567601621151 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23992082476615906 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10795783996582031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7015198469161987 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02444501966238022 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8597559332847595 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8146528601646423 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.659700870513916 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7038770914077759 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6321260929107666 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8841870427131653 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8079742193222046 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953141808509827 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867941737174988 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7511112093925476 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8618921637535095 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921292245388031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09314599633216858»Ÿ˜‹Õ=èÏ{ & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6234888434410095 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7465888261795044 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04772433638572693 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19670291244983673 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19704081118106842 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5675312876701355 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4966926574707031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2990580201148987 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 jet/accuracy_4_of_5‚0 % _timestamp‚1.7843464978549354e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2432282567024231 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28588372468948364 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6391420364379883 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22292514145374298 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942053556442261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840845465660095 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.378718763589859 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7104777693748474 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.563442587852478 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22569754719734192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7985072135925293 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7748858332633972 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2703286409378052 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_4‚0.9720036387443542 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6793110370635986 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2172321379184723 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6693258285522461 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9124894738197327 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 .particle/accuracy_3_of_4‚0.993424117565155 *particle/sensitivity‚0.623106837272644 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6374256610870361 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5701592564582825 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02600567601621151 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23992082476615906 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10795783996582031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7015198469161987 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02444501966238022 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8597559332847595 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8146528601646423 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.659700870513916 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7038770914077759 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6321260929107666 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8841870427131653 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8079742193222046 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953141808509827 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867941737174988 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7511112093925476 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8618921637535095 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921292245388031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09314599633216858 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.774986982345581 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02688428945839405 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7204279899597168 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6568119525909424 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6378474235534668 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44105952978134155 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7579284310340881 particle/accuracy_4_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7019141316413879 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.031910449266433716 +particle/specificity‚0.9047360420227051 jet/accuracy_3_of_5‚1 particle/accuracy_1_of_5‚1 epoch‚17 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.904312789440155 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7336767911911011 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34123745560646057 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982470273971558 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8044596314430237 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984246492385864 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34665948152542114 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22811663150787354 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1261780709028244 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02444501966238022 trainer/global_step‚7199 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6312649846076965 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02646058239042759 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_1_of_2‚0.7555528879165649 particle/accuracy_1_of_2‚1 *loss/val_total_loss‚7.9272891336445275 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36853286623954773 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6291236281394958 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22720254957675934 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5511418581008911 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.778626561164856 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6578007936477661 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237908005714417 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036031972616910934 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7795096635818481 +jet/accuracy_3_of_4‚0.08533445745706558 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.313079297542572 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2790018618106842 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7205245494842529 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8677479028701782 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8677479028701782 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_4_of_4‚0.9486338496208191 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7292895317077637 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025794360786676407 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6968379616737366 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11097162216901779 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.34395334124565125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6996828317642212 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7852985858917236 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28881460428237915 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_5_of_5‚0 _step‚593 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948212504386902 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848228693008423 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28881460428237915 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6780388951301575 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7804210186004639 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 )jet/accuracy_2_of_2‚0.155134379863739 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7847316265106201 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.611487865447998 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2135114222764969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954733848571777 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862183332443237 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29751646518707275 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2159487009048462 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45816153287887573 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038506485521793365 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37128788232803345 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6177191138267517 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5115148425102234 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06109023094177246 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4838956296443939 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6997324824333191 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25265732407569885 +jet/accuracy_3_of_3‚0.07130808383226395 _runtime‚6156.549218161 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48875975608825684 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9931118488311768 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5675630569458008 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07374705374240875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6234888434410095 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2966991364955902 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2319529801607132 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.313265860080719 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.658383846282959 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 (particle/accuracy‚0.8442537188529968 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.968169093132019 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094ÑKÁ9ìžé‚(:“ ŠïëÒüñ™” gpu.0.gpu‚27.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðò Ìê‚(:Á ŠïëÒÑ©– network.sent‚ 1.1384361e+07 disk.drbd2165.out‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 2723.984375" disk.dm-0.out‚22868.9033203125 disk.sda3.out‚ 193.15234375' proc.memory.availableMB‚ 494.8203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚22868.9033203125 disk.rbd1.out‚ 15.20703125) proc.memory.percent‚16.62588119506836 disk.rbd0.out‚0 disk.dm-1.out‚ 193.15234375 disk.drbd2165.in‚0 disk.dm-0.in‚ 12246.859375 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 12246.859375$ memory_percent‚96.97985649108887" disk.nvme0c0n1.in‚ 12246.859375 disk.sda.out‚ 193.15234375 disk.sda2.out‚0 cpu‚26.18298888388788 proc.cpu.threads‚17) disk./.usagePercent‚40.25171029417355$ disk./.usageGB‚599.7795715332031' disk.nvme0c0n1.out‚22868.9033203125 disk.rbd1.in‚0 network.recv‚ 1.685898e+06 disk.sda1.in‚0&Ÿ_žë‚(:“ ™ïëÒ½ª°” gpu.0.gpu‚63.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N•èáËì‚(:À ™ïëÒ¸§‘–( disk./.usagePercent‚40.2516957017441 disk.drbd2165.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.691007e+06 disk.dm-1.out‚ 193.4765625 disk.drbd2165.in‚0# disk.dm-0.out‚22875.06005859375' proc.memory.availableMB‚ 494.7890625! proc.memory.rssMB‚ 2723.984375) proc.memory.percent‚16.62588119506836 disk.sda1.out‚0& disk.nvme0n1.out‚22875.06005859375 disk.sda3.out‚ 193.4765625" disk.nvme0c0n1.in‚ 12246.859375 disk.dm-0.in‚ 12246.859375 disk.sda.out‚ 193.4765625 disk.sda2.out‚0 disk.nvme0n1.in‚ 12246.859375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.98004722595215# disk./.usageGB‚599.779354095459 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 15.20703125 network.sent‚ 1.1441738e+07( disk.nvme0c0n1.out‚22875.06005859375 cpu‚26.183311746764705Ë ³g‘í‚(:† ¨ïëÒÏæ×” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.272' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•çÁùÒî‚(:Ç ¨ïëÒî¥×• network.sent‚ 1.1445775e+07 disk.sda2.in‚0' proc.memory.availableMB‚ 494.6171875) proc.memory.percent‚16.62588119506836 disk.drbd2165.out‚0 disk.sda.in‚0 cpu‚26.183878180994032& disk.nvme0n1.out‚22882.45751953125 network.recv‚ 1.693563e+06 disk.sda3.out‚ 193.6796875 disk.rbd1.out‚ 15.26953125 disk.rbd0.in‚0# disk.dm-0.out‚22882.45751953125$ disk./.usageGB‚599.7791786193848 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# memory_percent‚96.9810962677002! proc.memory.rssMB‚ 2723.984375 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚12326.05859375 proc.cpu.threads‚17 disk.sda2.out‚0" disk.nvme0n1.in‚12326.05859375 disk.dm-1.out‚ 193.6796875$ disk.nvme0c0n1.in‚12326.05859375 disk.sda.out‚ 193.6796875) disk./.usagePercent‚40.25168392539752 disk.sda3.in‚0( disk.nvme0c0n1.out‚22882.45751953125íŸ2ï- _wandbruntime‚6196  _runtime‚6196`´‡ÇðÁ & lr-AdamW/pg1‚0.0003268293882888854 & lr-AdamW/pg2‚0.0003268293882888854 trainer/global_step‚7249 % _timestamp‚1.7843465378427563e+09 _runtime‚6196.536331916 _step‚594Ú.|ÌñÆ & lr-AdamW/pg1‚0.0003268293882888854 & lr-AdamW/pg2‚0.0003268293882888854 trainer/global_step‚7249 % _timestamp‚1.7843465378427563e+09 _runtime‚6196.536331916 _step‚594Òñ©y2ò- _wandbruntime‚6196  _runtime‚6196at÷ŸóíO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57507848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250520445252096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67168158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7658936296434 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124676008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28301677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57507850 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7670636471296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67168158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 6196.53756832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1102265488482304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670977810432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242768896002048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670977810432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1102265505521664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15598912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1096593592741888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109936483332096 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1102265488482304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1109936483332096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7751464867328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57507848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ Ã(†žT 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104252511998962 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1109936124953600 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67158181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1096593575702528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67158181 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104252869588212 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124666029 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109936124953600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28301400 trainer/global_step‚7249 _step‚595 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169Æ€%¾ø'ôòO % _timestamp‚1.7843465378438292e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7670636471296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250520342257152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1102265505521664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242768877389824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1096593592741888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109936483332096 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1102265488482304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1109936483332096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7751464867328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57507848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104252511998962 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1109936124953600 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67158181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1096593575702528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67158181 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104252869588212 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124666029 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109936124953600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28301400 trainer/global_step‚7249 _step‚595 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57507848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250520445252096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67168158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7658936296434 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124676008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28301677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57507850 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7670636471296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67168158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 6196.53756832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1102265488482304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670977810432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242768896002048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670977810432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1102265505521664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15598912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1110084736995840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671990444544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7752573739008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 6197.15347489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28305236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15601003 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671990444544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7659947350156 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚596 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1102412746551296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57515480 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67167233 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1102412729511936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12704236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15601273 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28305509 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1102412729511936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242802306148352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1110084736995840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚7249Úû¢•b%÷üM _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1096740103413760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1102412746551296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124682713 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250554795511808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28305236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15601003 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671990444544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7659947350156 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚596 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1102412746551296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57515480 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67167233 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1102412729511936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12704236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15601273 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28305509 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1102412729511936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242802306148352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1110084736995840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚7249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7671649098240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67177224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7752489363456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1110084736995840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671990444544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7752573739008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 6197.15347489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_alê,ü locations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15702615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7709552783342 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1109627539963904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252243004549632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67612633 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚6236.920455434 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1117349230315520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843465782265098e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109627557003264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57889448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28492624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1109627557003264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1109627539963904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67612633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7721673312256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252243111738880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1117348871937024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7721673312256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15702883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57889448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _step‚599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12789738 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7721331973120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57889450 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57889450 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1103919126335488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1117349230315520 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1117348871937024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67622610 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67622610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7802689233408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ê®Éù'„óO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1109627557003264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1109627539963904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67612633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7721673312256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252243111738880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1117348871937024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7721673312256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15702883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57889448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _step‚599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12789738 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7721331973120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57889450 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57889450 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1103919126335488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1117349230315520 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1117348871937024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67622610 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67622610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7802689233408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1103919109296128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244440315316224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7802777810432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15702615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7709552783342 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1109627539963904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252243004549632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67612633 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚6236.920455434 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1117349230315520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843465782265098e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109627557003264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57889448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28492624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0×oeŸ12…- _wandbruntime‚6237  _runtime‚6237‰²_ІøM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528758 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67631676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67631676 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57897080 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1109774798032896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244473762686976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104065619968000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28496198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚7299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67621685 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1117497483979264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7722344600064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125518765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12791486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125528758 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125518765 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7710904387066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1117497125593600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7722344600064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154î^ƒ.  LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 % _timestamp‚1.7843465788455656e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1109774780993536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1109774798032896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7722685946368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7803796228608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7803703464448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244473744074752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1109774780993536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252277447539200 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104065637007360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252277558915584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111776541394426 _runtime‚6237.539485187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57897082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1117497483979264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144tCÉ°'‡ýM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125518765 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7710904387066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1117497125593600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7722344600064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 % _timestamp‚1.7843465788455656e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1109774780993536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1109774798032896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7722685946368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7803796228608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7803703464448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244473744074752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1109774780993536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252277447539200 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104065637007360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252277558915584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111776541394426 _runtime‚6237.539485187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57897082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1117497483979264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528758 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67631676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67631676 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57897080 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1109774798032896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244473762686976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104065619968000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28496198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚7299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67621685 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1117497483979264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7722344600064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125518765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12791486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125528758 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246111771854848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚603 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58271050 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126338133 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68077062 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7853968253952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15806726 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7760169270250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68067085 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58271048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7772368814080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1124761977298944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7772368814080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7853879676928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68067085 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68077062 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7772027474944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253965740108800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12876988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1124761977298944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58271050 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7349 $ _timestamp‚1.784346619259969e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚6277.953857839 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124761618920448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116989591445504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760509820140 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1116989608484864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124761618920448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7853879676928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68067085 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68077062 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7772027474944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253965740108800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12876988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1124761977298944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58271050 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7349 $ _timestamp‚1.784346619259969e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚6277.953857839 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124761618920448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor¯§@î.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116989591445504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760509820140 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1116989608484864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124761618920448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246111771854848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚603 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58271050 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126338133 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68077062 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7853968253952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15806726 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7760169270250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68067085 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58271048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÛzMTö2—- _wandbruntime‚6278  _runtime‚6278¬'Îõþ&˜øM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68076137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1117136832475136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773040101888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15808534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246145182001152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111391153561600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784346619876391e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126364810 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68086128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124909872577024 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117136849514496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58278680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254000195027456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7773040101888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12878736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7854994414080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117136849514496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15808809 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28687267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12878733 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773381448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58278682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58278682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7854893261312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1119152691474934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58278680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126354817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119152333885572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28687545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7773381448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1117136832475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1124910230962688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7349 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1124910230962688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0]Á'™ýM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58278682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7854893261312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1119152691474934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58278680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126354817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119152333885572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28687545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7773381448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1117136832475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1124910230962688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7349 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1124910230962688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68076137 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1117136832475136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773040101888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15808534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246145182001152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111391153561600 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784346619876391e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126364810 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68086128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124909872577024 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117136849514496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58278680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254000195027456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7773040101888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12878736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7854994414080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117136849514496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15808809 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28687267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12878733 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773381448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58278682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438Üø¹ˆ2š- _wandbruntime‚6278  _runtime‚6278Ór,¤ ›„ _step‚605 2loss/FBt2/assignment_loss‚0.001523379257745776 %loss/total_loss‚7.495147254491685 0loss/FBt2/detection_loss‚0.36891263362923565 /loss/FRt1/assignment_loss‚1.599605733726228 3loss/SRqqt1/assignment_loss‚0.47560938070162356 1loss/FBt1/assignment_loss‚0.03756315685402584 1loss/SRqqt2/detection_loss‚0.6113518139752109 $ _timestamp‚1.784346619880214e+09 0loss/FRt2/assignment_loss‚0.7772653123870521 /loss/FRt1/detection_loss‚0.5616012859970062 epoch‚18 3loss/SRqqt2/assignment_loss‚0.49811479432149525 /loss/FRt2/detection_loss‚1.1587766824874968 1loss/SRqqt1/detection_loss‚0.8657348590894226 /loss/FBt1/detection_loss‚0.5390882220651422 trainer/global_step‚7349 _runtime‚6278.573507246Åsœ‰ _runtime‚6278.573507246 _step‚605 2loss/FBt2/assignment_loss‚0.001523379257745776 %loss/total_loss‚7.495147254491685 0loss/FBt2/detection_loss‚0.36891263362923565 /loss/FRt1/assignment_loss‚1.599605733726228 3loss/SRqqt1/assignment_loss‚0.47560938070162356 1loss/FBt1/assignment_loss‚0.03756315685402584 1loss/SRqqt2/detection_loss‚0.6113518139752109 $ _timestamp‚1.784346619880214e+09 0loss/FRt2/assignment_loss‚0.7772653123870521 /loss/FRt1/detection_loss‚0.5616012859970062 epoch‚18 3loss/SRqqt2/assignment_loss‚0.49811479432149525 /loss/FRt2/detection_loss‚1.1587766824874968 1loss/SRqqt1/detection_loss‚0.8657348590894226 /loss/FBt1/detection_loss‚0.5390882220651422 trainer/global_step‚7349Ý” cA“‚(:ˆ ‚ðëÒÖÐÚ” gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aš/Âìž‚(:á ‚ðëÒ£¥– disk.dm-1.in‚ 0.03515625( disk.nvme0c0n1.out‚22937.97998046875' proc.memory.availableMB‚ 494.7890625# disk.nvme0c0n1.in‚ 12407.2890625) disk./.usagePercent‚40.25170133390985 disk.sda.out‚ 197.10546875 disk.sda2.out‚0! disk.nvme0n1.in‚ 12407.2890625& disk.nvme0n1.out‚22937.97998046875 disk.rbd0.in‚0) proc.memory.percent‚16.62588119506836 disk.dm-0.in‚ 12407.2890625 disk.sda3.in‚ 0.03515625 disk.rbd1.in‚0 disk.dm-1.out‚ 197.10546875 disk.drbd2165.in‚0 cpu‚26.1837859123146 proc.cpu.threads‚17 disk.sda.in‚ 0.03515625$ memory_percent‚96.98004722595215! proc.memory.rssMB‚ 2723.984375 disk.sda2.in‚0$ disk./.usageGB‚599.7794380187988 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.sent‚ 1.1591873e+07 network.recv‚ 1.717864e+06# disk.dm-0.out‚22937.97998046875 disk.sda3.out‚ 197.10546875 disk.rbd1.out‚ 15.45703125"£!žŸ‚(:“ ‘ðëÒæô´“ gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³<›å ‚(:Ú ‘ðëÒíþ–" disk.nvme0n1.in‚12408.29296875 disk.rbd1.out‚ 15.45703125 disk.dm-1.in‚ 0.03515625 disk.dm-1.out‚ 201.7421875$ disk.nvme0n1.out‚22961.822265625 disk.sda2.in‚0) disk./.usagePercent‚40.25168162132971$ memory_percent‚96.98123931884766% proc.memory.availableMB‚ 494.59375 cpu‚26.183304843943404 disk.sda.in‚ 0.03515625 disk.sda2.out‚0 network.recv‚ 1.72311e+06 disk.sda3.in‚ 0.03515625 disk.sda3.out‚ 201.7421875 disk.sda1.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚22961.822265625 disk.rbd1.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚22961.822265625 disk.sda.out‚ 201.7421875! proc.memory.rssMB‚ 2723.984375) proc.memory.percent‚16.62588119506836 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚12408.29296875 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 1.1654819e+07$ disk./.usageGB‚599.7791442871094 disk.dm-0.in‚12408.29296875 proc.cpu.threads‚17Òºž¡‚(:“  ðëÒÐꉕ gpu.0.gpu‚98.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚226.004' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.200799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vú¢j™¢‚(:Þ  ðëÒ›É÷•( disk.nvme0c0n1.out‚22968.55322265625 network.recv‚ 1.726577e+06 disk.sda1.in‚0 disk.dm-1.out‚ 205.546875 disk.drbd2165.in‚0 disk.dm-0.in‚ 12509.8359375& disk.nvme0n1.out‚22968.55322265625) proc.memory.percent‚16.62588119506836 disk.drbd2165.out‚0 disk.sda3.out‚ 205.546875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1659937e+07) disk./.usagePercent‚40.25167624517149 disk.sda1.out‚0 disk.dm-1.in‚ 0.03515625# disk.dm-0.out‚22968.55322265625 disk.sda2.in‚0 disk.sda3.in‚ 0.03515625( proc.memory.availableMB‚ 494.59765625$ disk./.usageGB‚599.ü?ëP7790641784668 disk.sda.out‚ 205.546875! disk.nvme0n1.in‚ 12509.8359375$ memory_percent‚96.98121547698975 disk.sda.in‚ 0.03515625 disk.rbd1.out‚ 15.58203125 disk.rbd0.out‚0! proc.memory.rssMB‚ 2723.984375 cpu‚26.182841060081525# disk.nvme0c0n1.in‚ 12509.8359375 disk.sda2.out‚0 disk.rbd0.in‚0„5›b2£- _wandbruntime‚6318  _runtime‚6318Ñ8ãˆÉ¤Ã trainer/global_step‚7399 % _timestamp‚1.7843466600188098e+09 _runtime‚6318.715573813 _step‚606 ' lr-AdamW/pg1‚0.00032199448766554595 ' lr-AdamW/pg2‚0.00032199448766554595‘õ<0Î¥È ' lr-AdamW/pg1‚0.00032199448766554595 ' lr-AdamW/pg2‚0.00032199448766554595 trainer/global_step‚7399 % _timestamp‚1.7843466600188098e+09 _runtime‚6318.715573813 _step‚606Þ¬0½2¦- _wandbruntime‚6318  _runtime‚6318á2‹õ'§ïO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7823064315904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7905041486336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58652650 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28874596 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1126380962240486 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118570176483328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124351659966464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811126307048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843466600227237e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28874301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1126381319829736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1132174724282368 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124351642927104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7822722976768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12964238 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1132174365903872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255688232655360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68531514 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15910355 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174185 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1132174365903872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68521537 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7822722976768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58652648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58652650 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68531514 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255688354524672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103257088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7810785757158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1124351659966464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12964241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7905144743424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124351659966464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811126307048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843466600227237e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28874301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1126381319829736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1132174724282368 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124351642927104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7822722976768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12964238 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1132174365903872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255688232655360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68531514 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15910355 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174185 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1132174365903872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68521537 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7822722976768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58652648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58652650 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68531514 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255688354524672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103257088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7810785757158 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1124351659966464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ß- Ý2©- _wandbruntime‚6319  _runtime‚6319½Ýü&ªöM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚ 6319.3335906 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7824076950016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68540580 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7823735603712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7811796810880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7823735603712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15912396 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190869 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200862 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1124498883956736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118716687155200 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1132322977946112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1124498883956736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68530589 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7906157464576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843466606397169e+09 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28878107 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965986 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7906066797568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1126528841555442 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118716704194560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58660280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58660282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚7399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127200862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12965983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68530589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28878107 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965986 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7906066797568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1126528841555442 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118716704194560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58660280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58660282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚7399 L>DeviceStatsÑbm™¯!Monitor.on_train_batch_end/allocation.all.allocated‚ 127200862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12965983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68530589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68540580 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7823735603712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7811796810880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7823735603712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15912396 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190869 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200862 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1124498883956736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118716687155200 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1132322977946112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1124498883956736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68530589 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7906157464576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843466606397169e+09 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1133757112320994 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1133757469910244 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1139587112887296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128020216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125895710076928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59034248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13051491 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1131713694408704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257410778214400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861402244066 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131713711448064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68985966 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249454647707648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1139587471265792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29064726 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚611 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131713694408704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7873418478592 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1139587471265792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59034248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1131713711448064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59034250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843467009768958e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249454629095424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68975989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¯Ä+±"¸óO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1133757112320994 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1133757469910244 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1139587112887296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128020216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125895710076928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59034248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13051491 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1131713694408704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257410778214400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861402244066 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131713711448064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68985966 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249454647707648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1139587471265792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29064726 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚611 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131713694408704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7873418478592 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1139587471265792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59034248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_by™ßHtes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1131713711448064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59034250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843467009768958e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249454629095424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68975989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376ã«Ÿ¡Á‘¹‚(:† ÍðëҦΓ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì2ªáº‚(:Ö ÍðëÒƒîì– disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 cpu‚26.181310592626357 network.recv‚ 1.739333e+06$ disk./.usageGB‚599.7785797119141$ memory_percent‚96.98116779327393( proc.memory.availableMB‚ 494.60546875 disk.sda1.in‚0 disk.dm-0.in‚12511.64453125 disk.rbd1.out‚ 15.64453125 network.sent‚ 1.1733038e+07" disk.dm-0.out‚22994.4208984375 proc.cpu.threads‚17 disk.dm-1.out‚208.3125 disk.sda.out‚208.3125 disk.sda2.in‚0" disk.nvme0n1.in‚12511.64453125% disk.nvme0n1.out‚22994.4208984375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚12511.64453125 disk.sda3.out‚208.3125) proc.memory.percent‚16.62588119506836* disk./.usagePercent‚40.251643732214646 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚22994.4208984375! proc.memory.rssMB‚ 2723.984375øÊeî2»- _wandbruntime‚6360  _runtime‚6360îP‡þ&¼øM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128026921 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7874772451840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68985041 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1139735724929536 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1139735366543872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249488076466176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1131860935438336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1131860935438336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1126042237788160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1139735724929536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7874431105536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1131860952477696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59041882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257445338348032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29068196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13053233 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68985041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16015240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68995032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1126042220748800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7874772451840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7874431105536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59041880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1131860952477696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036914 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843467015947237e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59041880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7957261881856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59041882 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249488057853952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1133904991635950 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚6360.288867506 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16014963 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976ëÇÁœ'½ýM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7874772451840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68985041 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1139735724929536 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1139735366543872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249488076466176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1131860935438336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1131860935438336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1126042237788160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1139735724929536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7874431105536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1131860952477696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59041882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257445338348032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29068196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13053233 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68985041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16015240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68995032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1126042220748800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7874772451840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7874431105536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59041880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1131860952477696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036914 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843467015947237e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59041880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7957261881856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59041882 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249488057853952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1133904991635950 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚6360.288867506 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16014963 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128026921ädSÎ12¾- _wandbruntime‚6360  _runtime‚6360½t2 ¿„ /loss/FBt1/detection_loss‚0.8718858585836222 _runtime‚6360.291573245 0loss/FRt1/assignment_loss‚1.6504748033965293 0loss/FRt2/assignment_loss‚0.8004047786873858 2loss/SRqqt1/assignment_loss‚0.6462616610766881 2loss/SRqqt2/assignment_loss‚0.5098311549110686 /loss/FBt2/detection_loss‚0.6301727795338609 trainer/global_step‚7449 % _timestamp‚1.7843467015982783e+09 2loss/FBt2/assignment_loss‚0.003955612176513154 /loss/FRt1/detection_loss‚0.5698131290183187 /loss/FRt2/detection_loss‚1.2827866142661317 epoch‚18 2loss/FBt1/assignment_loss‚0.003752655238955208 1loss/SRqqt1/detection_loss‚0.8782422525258172 %loss/total_loss‚8.447531656716638 _step‚613 1loss/SRqqt2/detection_loss‚0.5999503573017456²‚E(À‰ _runtime‚6360.291573245 0loss/FRt1/assignment_loss‚1.6504748033965293 0loss/FRt2/assignment_loss‚0.8004047786873858 2loss/SRqqt1/assignment_loss‚0.6462616610766881 2loss/SRqqt2/assignment_loss‚0.5098311549110686 /loss/FBt2/detection_loss‚0.6301727795338609 trainer/global_step‚7449 % _timestamp‚1.7843467015982783e+09 2loss/FBt2/assignment_loss‚0.003955612176513154 /loss/FRt1/detection_loss‚0.5698131290183187 /loss/FRt2/detection_loss‚1.2827866142661317 epoch‚18 2loss/FBt1/assignment_loss‚0.003752655238955208 1loss/SRqqt1/detection_loss‚0.8782422525258172 %loss/total_loss‚8.447531656716638 _step‚613 1loss/SRqqt2/detection_loss‚0.5999503573017456 /loss/FBt1/detection_loss‚0.8718858585836222åÐp¦É’Á‚(:‡ ÜðëÒ”§Î“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.201 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/¢©¼æ‚(:Û ÜðëÒ»®¤–' disk.nvme0c0n1.out‚23001.3466796875$ memory_percent‚96.98293209075928 disk.sda1.out‚0 disk.dm-1.out‚ 208.79296875 disk.sda.out‚ 208.79296875 disk.rbd1.out‚ 15.64453125 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 network.sent‚ 1.1785294e+07 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0! proc.memory.rssMB‚ 2723.984375 disk.dm-0.in‚ 12580.296875 disk.rbd0.out‚0) proc.memory.percent‚16.62588119506836 cpu‚26.18085167386333 network.recv‚ 1.744133e+06 disk.drbd2165.in‚0" disk.dm-0.out‚23001.3466796875$ disk./.usageGB‚599.7786331176758" disk.nvme0c0n1.in‚ 12580.296875 disk.sda2.in‚0 disk.nvme0n1.in‚ 12580.296875 disk.sda3.out‚ 208.79296875( proc.memory.availableMB‚ 494.31640625 disk.drbd2165.out‚0% disk.nvme0n1.out‚23001.3466796875* disk./.usagePercent‚40.251647316320124 disk.dm-1.in‚ 0.1328125ªÂ’‘Â(:† ëðëÒ‘ææ“ gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±æ àÄ‚(:Õ ëðëÒχ‡–) disk./.usagePercent‚40.24833099472158 disk.sda3.out‚ 209.73046875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.out‚23006.77734375( proc.memory.availableMB‚ 494.50390625 cpu‚26.180781441631282 network.sent‚ 1.1837765e+07 network.recv‚ 1.749438e+06 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 15.70703125) proc.memory.percent‚16.62588119506836 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 209.73046875% disk.nvme0c0n1.out‚23006.77734375 disk.dm-0.in‚ 12580.296875! proc.memory.rssMB‚ 2723.984375$ disk./.usageGB‚599.7292175292969# disk.nvme0n1.out‚23006.77734375 disk.rbd1.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 12580.296875 disk.nvme0n1.in‚ 12580.296875 disk.drbd2165.in‚0 disk.sda.out‚ 209.73046875$ memory_percent‚96.98178768157959 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚ 0.1328125zÚœg2Å- _wandbruntime‚6400  _runtime‚6400™æÍÆÆÀ & lr-AdamW/pg1‚0.0003187397509635105 & lr-AdamW/pg2‚0.0003187397509635105 trainer/global_step‚7499 $ _timestamp‚1.784346741786767e+09 _runtime‚6400.480354485 _step‚614ËgŽ,ËÇÅ trainer/global_step‚7499 $ _timestamp‚1.784346741786767e+09 _runtime‚6400.480354485 _step‚614 & lr-AdamW/pg1‚0.0003187397509635105 & lr-AdamW/pg2‚0.0003187397509635105æø.Ð2È- _wandbruntime‚6400  _runtime‚6400´Bi,lÉîO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7912018730974 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1139075762929664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1139075762929664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13138738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69440418 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8007256206336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7912359280864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚7499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856268 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259133434611712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1141133262401502 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59415850 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1133221260709888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69430441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259133323228160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8007348977664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146999859870720 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139075745890304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1147000218249216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_bat‘®-Uˆch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59415848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69440418 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1147000218249216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128846289 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1146999859870720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29255163 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16116422 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7924455319552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1141133619990752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7924113980416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69430441 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59415850 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7924113980416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29254882 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59415848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251126067021824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1133221243670528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13138741 _runtime‚6400.481598332 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7924455319552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16116144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843467417878475e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251126085634048såˆåù'ÊóO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7924113980416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29254882 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59415848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251126067021824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1133221243670528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13138741 _runtime‚6400.481598332 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7924455319552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16116144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843467417878475e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251126085634048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7912018730974 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1139075762929664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1139075762929664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13138738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69440418 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8007256206336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7912359280864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚7499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856268 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259133434611712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1141133262401502 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59415850 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1133221260709888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69430441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259133323228160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8007348977664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146999859870720 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139075745890304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1147000218249216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59415848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69440418 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1147000218249216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128846289 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1146999859870720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29255163 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16116422 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7924455319552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1141133619990752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7924113980416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69430441 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59415850ç\R)A2Ë- _wandbruntime‚6401  _runtime‚6401Â-šØÿ&ÌùM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚7499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13140483 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872966 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259167884287488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29258684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251159514392576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1139223003959296 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59423482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59423482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8008266644992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862973 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259167762425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139223003959296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1133367754342400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1139222986919936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13140486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251159495780352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7913029784696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7925126607360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1147148471912960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1141281141716458 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1139222986919936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚6401.094976369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7925126607360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16118201 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1133367771381760 % _timestamp‚1.7843467424010558e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1147148471912960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1147148113527296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69439493 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29258966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59423480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8008369894912hãƒØ+ÍþM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1133367771381760 % _timestamp‚1.7843467424010558e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1147148471912960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1147148113527296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69439493 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29258966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59423480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8008369894912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚7499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13140483 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872966 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259167884287488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29258684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251159514392576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1139223003959296 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segme»m?Ù nt.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59423482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59423482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8008266644992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862973 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259167762425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139223003959296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1133367754342400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1139222986919936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13140486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251159495780352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7913029784696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7925126607360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1147148471912960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1141281141716458 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1139222986919936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚6401.094976369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7925126607360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16118201èÀË‚ž2Î- _wandbruntime‚6401  _runtime‚6401c›óŽÏ€ 2loss/SRqqt1/assignment_loss‚0.5391269665253937 2loss/SRqqt2/assignment_loss‚0.4740793245943441 /loss/FBt2/detection_loss‚0.4784548526076833 % _timestamp‚1.7843467424048707e+09 _runtime‚ 6401.09821246 0loss/FRt2/assignment_loss‚0.8312491723806551 1loss/SRqqt1/detection_loss‚0.8584902687580325 %loss/total_loss‚8.063845656152303 0loss/FRt1/assignment_loss‚1.5525393035690702 2loss/FBt1/assignment_loss‚0.003922800452367291 /loss/FRt2/detection_loss‚1.2745120926830447 -loss/FBt1/detection_loss‚0.73509982275981 1loss/FBt2/assignment_loss‚0.09431129299852542 /loss/FRt1/detection_loss‚0.6024306920957165 1loss/SRqqt2/detection_loss‚0.6196290667276594 epoch‚18 trainer/global_step‚7499 _step‚617 À3 Ð… trainer/global_step‚7499 _step‚617 2loss/SRqqt1/assignment_loss‚0.5391269665253937 2loss/SRqqt2/assignment_loss‚0.4740793245943441 /loss/FBt2/detection_loss‚0.4784548526076833 % _timestamp‚1.7843467424048707e+09 _runtime‚ 6401.09821246 0loss/FRt2/assignment_loss‚0.8312491723806551 1loss/SRqqt1/detection_loss‚0.8584902687580325 %loss/total_loss‚8.063845656152303 0loss/FRt1/assignment_loss‚1.5525393035690702 2loss/FBt1/assignment_loss‚0.003922800452367291 /loss/FRt2/detection_loss‚1.2745120926830447 -loss/FBt1/detection_loss‚0.73509982275981 1loss/FBt2/assignment_loss‚0.09431129299852542 /loss/FRt1/detection_loss‚0.6024306920957165 1loss/SRqqt2/detection_loss‚0.6196290667276594 epoch‚18éY´¨“‘Ñ‚(:† úðëÒºîÜ” gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕQìÒ‚(:á úðëÒÊ í– network.sent‚ 1.1842733e+07$ disk.nvme0c0n1.in‚12581.26171875 disk.sda2.in‚0 disk.sda1.in‚0# disk.dm-0.out‚23013.89501953125 disk.sda.out‚ 210.2421875& disk.nvme0n1.out‚23013.89501953125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 210.2421875! proc.memory.rssMB‚ 2723.984375( disk.nvme0c0n1.out‚23013.89501953125 disk.sda.in‚ 0.1328125 disk.drbd2165.in‚0) proc.memory.percent‚16.62588119506836 proc.cpu.threads‚17" disk.nvme0n1.in‚12581.26171875$ memory_percent‚96.98488712310791( proc.memory.availableMB‚ 493.99609375 cpu‚26.180236690178706 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0) disk./.usagePercent‚40.24832280248048 disk.dm-0.in‚12581.26171875 disk.sda2.out‚0 network.recv‚ 1.752731e+06$ disk./.usageGB‚599.7290954589844 disk.rbd1.out‚ 15.70703125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 210.2421875 disk.rbd1.in‚0 disk.sda1.out‚01üÓ‚(:’ ‰ñëÒô„– gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÛRqæÔ‚(:Û ‰ñëÒ“Ȥ˜$ memory_percent‚96.98302745819092 cpu‚26.179766772028525! disk.nvme0c0n1.in‚ 12582.09375( disk.nvme0c0n1.out‚23023.45849609375 disk.dm-0.in‚ 12582.09375 disk.sda3.out‚ 210.50390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚23023.45849609375 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 494.30078125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 12582.09375 proc.cpu.threads‚17* disk./.usagePercent‚40.248320242405136 disk.rbd1.out‚ 15.76953125$ disk./.usageGB‚599.7290573120117 disk.sda2.in‚0& disk.nvme0n1.out‚23023.45849609375 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 network.recv‚ 1.75834e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 210.50390625! proc.memory.rssMB‚ 2723.984375) proc.memory.percent‚16.62588119506836 network.sent‚ 1.1906147e+07 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.out‚ 210.50390625 disk.sda2.out‚0ÓÛö)‘Õ‚(:† ˜ñëÒ´¾ä” gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÄÍÝÖ‚(:Ò ˜ñëÒŽ¯“– disk.rbd1.in‚0 disk.rbd1.out‚ 15.76953125$ memory_percent‚96.98295593261719 proc.cpu.threads‚17 network.recv‚ 1.761708e+06) disk./.usagePercent‚40.24831717031472 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 210.78515625$ proc.memory.availableMB‚494.3125$ disk./.usageGB‚599.7290115356445 disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚ 1.1911007e+07 disk.dm-0.in‚ 12667.8125 disk.sda2.in‚0 disk.nvme0n1.in‚ 12667.8125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 12667.8125( disk.nvme0c0n1.out‚23025.97607421875# disk.dm-0.out‚23025.97607421875 disk.sda2.out‚0) proc.memory.percent‚16.62588119506836 disk.dm-1.out‚ 210.78515625 disk.drbd2165.in‚0& disk.nvme0n1.out‚23025.97607421875! proc.memory.rssMB‚ 2723.984375 cpu‚26.1792127100487 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 210.78515625 disk.rbd0.in‚0 disk.sda3.in‚ 0.1328125”^2×- _wandbruntime‚6441  _runtime‚6441"á#ÆØÀ $ _timestamp‚1.784346782783798e+09 _runtime‚6441.477352823 _step‚618 & lr-AdamW/pg1‚0.0003171032546940039 & lr-AdamW/pg2‚0.0003171032546940039 trainer/global_step‚7549ŒØªËÙÅ & lr-AdamW/pg1‚0.0003171032546940039 & lr-AdamW/pg2‚0.0003171032546940039 trainer/global_step‚7549 $ _timestamp‚1.784346782783798e+09 _runtime‚6441.477352823 _step‚618ê`[ 2Ú- _wandbruntime‚6441  _runtime‚6441Éç>±ô'ÛîO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146437797371904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7974809482240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59797448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29444500 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7975150821376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29444777 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1154412965232640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146437814411264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13225991 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69894870 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8058235211776 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1148509412482010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13225988 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843467827848575e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129682341 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69884893 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59797450 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69894870 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146437814411264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129692320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16218512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252797523560448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252797504948224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6441.478617551 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260855857835008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1148509770071260 _step‚619 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1146437797371904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260855740160000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8058334274560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7974809482240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1140546794303488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59797450 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59797448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1154412606854144Ðë ù'ÜóO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7975150821376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29444777 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1154412965232640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146437814411264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13225991 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69894870 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8058235211776 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1148509412482010 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13225988 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843467827848575e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129682341 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69884893 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59797450 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69894870 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146437814411264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129692320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16218512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252797523560448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252797504948224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6441.478617551 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260855857835008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1148509770071260 _step‚619 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1146437797371904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260855740160000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8058334274560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7974809482240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1140546794303488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59797450 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59797448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1154412606854144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146437797371904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7974809482240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59797448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29444500 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269뽑 2Ý- _wandbruntime‚6442  _runtime‚6442ñ‰jìä ÞøM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59805082 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709018 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16220534 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1148656934207604 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7975822109184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29448267 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252830933706752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69893945 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13227736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16220805 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7963646271604 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7975822109184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260890299367936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚6442.093340612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260890190088704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101v×DY'ßýM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1154561218896384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29448541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1154560860510720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59805080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69893945 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1154561218896384 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1146585038401536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59805080 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1140693304975360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146585055440896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚7549 % _timestamp‚1.7843467833994377e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129709018 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13227733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7963986821606 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7976163455488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252830952318976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8059256381952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709018 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16220534 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1148656934207604 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7975822109184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29448267 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252830933706752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69893945 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13227736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16220805 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7963646271604 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7975822109184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260890299367936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚6442.093340612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260890190088704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59805082 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8013251704790 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70349322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1161825353837568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70339345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8013592254680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179050 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚7599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8025504984064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025504984064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254468961486848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚6482.469603069 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025846323200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16320555 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1153799848853504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528372 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_st‹,)art/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1161825353837568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262578255364608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8013251704790 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70349322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1161825353837568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70339345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8013592254680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179050 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚7599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8025504984064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025504984064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254468961486848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚6482.469603069 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025846323200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16320555 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1153799848853504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528372 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1161825353837568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262578255364608 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1148018838568960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1161973607494144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1153947106922496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130545070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16322829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16322562 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8110307160064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535077 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13314983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1153947106922496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29637815 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262612697405440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1153947089883136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254502371633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014603308514 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29637545 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70348397 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545070 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70348397 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60186680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8026858957312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254502390245376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8026858957312°îsn'ñýM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1148018838568960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1161973607494144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1153947106922496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130545070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16322829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16322562 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8110307160064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535077 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13314983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1153947106922496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29637815 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262612697405440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1153947089883136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254502371633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014603308514 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29637545 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70348397 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545070 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70348397 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60186680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8026858957312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254502390245376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8026858957312 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1161973965879808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8026517611008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1161973965879808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544ðIæÌÔ2ò- _wandbruntime‚6483  _runtime‚64835ÃØM ó† 0loss/FRt1/assignment_loss‚1.5282200113460342 0loss/FRt2/assignment_loss‚0.8288732530867181 2loss/FBt2/assignment_loss‚0.004698056001541539 /loss/FRt2/detection_loss‚1.2702735542772599 trainer/global_step‚7599 _step‚625 3loss/FBt1/assignment_loss‚0.0002701448266136148 /loss/FRt1/detection_loss‚0.5993110416601419 1loss/SRqqt2/detection_loss‚0.6372202274007172 %loss/total_loss‚7.989752535124801 % _timestamp‚1.7843468243926594e+09 2loss/SRqqt1/assignment_loss‚0.5231361552789849 3loss/SRqqt2/assignment_loss‚0.48108051217314496 /loss/FBt2/detection_loss‚0.4137656108066704 1loss/SRqqt1/detection_loss‚0.8272006418405276 /loss/FBt1/detection_loss‚0.8757033264264464 epoch‚18 _runtime‚6483.085981212nmC,ô‹ 0loss/FRt1/assignment_loss‚1.5282200113460342 0loss/FRt2/assignment_loss‚0.8288732530867181 2loss/FBt2/assignment_loss‚0.004698056001541539 /loss/FRt2/detection_loss‚1.2702735542772599 trainer/global_step‚7599 _step‚625 3loss/FBt1/assignment_loss‚0.0002701448266136148 /loss/FRt1/detection_loss‚0.5993110416601419 1loss/SRqqt2/detection_loss‚0.6372202274007172 %loss/total_loss‚7.989752535124801 % _timestamp‚1.7843468243926594e+09 2loss/SRqqt1/assignment_loss‚0.5231361552789849 3loss/SRqqt2/assignment_loss‚0.48108051217314496 /loss/FBt2/detection_loss‚0.4137656108066704 1loss/SRqqt1/detection_loss‚0.8272006418405276 /loss/FBt1/detection_loss‚0.8757033264264464 epoch‚18 _runtime‚6483.085981212ñEÛÒõ‚(:ˆ ÔñëÒ¿þ­” gļBºupu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚191.313' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.2626 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p <Ñö‚(:Æ ÔñëÒ‘À– disk.sda.out‚ 211.953125 disk.rbd0.in‚0) proc.memory.percent‚16.71454906463623 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 12756.859375 disk.sda3.out‚ 211.953125 proc.cpu.threads‚17 network.sent‚ 1.198907e+07 network.recv‚ 1.7772e+06 disk.dm-0.in‚ 12756.859375 disk.sda.in‚ 0.1328125$ memory_percent‚97.07403182983398$ disk./.usageGB‚599.7303161621094 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.dm-1.out‚ 211.953125 disk.rbd1.out‚ 15.95703125# proc.memory.rssMB‚ 2738.51171875& proc.memory.availableMB‚ 479.390625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 12756.859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚26.184535957579968) disk./.usagePercent‚40.24840472489144! disk.nvme0n1.out‚ 23055.796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.out‚ 23055.796875 disk.dm-0.out‚ 23055.796875üåfÔ2÷- _wandbruntime‚6501  _runtime‚6501k¹mç=øá{ ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518066167831421 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19079573452472687 particle/accuracy_3_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025080248713493347 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47483325004577637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4887025058269501 particle/accuracy_4_of_5‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.776465892791748 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03490466624498367 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751248240470886 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚7.905027469348963 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8950256109237671 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_1_of_1‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5020833611488342 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2822962701320648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919931292533875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933842420578003 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8787442445755005 +jet/accuracy_3_of_3‚0.06283833086490631 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5649901628494263 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955137372016907 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6341164708137512 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7297318577766418 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6817772388458252 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6337522864341736 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2823989987373352 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3747287690639496 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996301531791687 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962449073791504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024475622922182083 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18724672496318817 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.804215669631958 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.897014319896698 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.049152832478284836 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9550725817680359 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1949562132358551 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.634980320930481 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7394860982894897 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3216533064842224 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20820482075214386 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31449222564697266 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867221117019653 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_3‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946061372756958 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_4‚0.7811702489852905 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2106267809867859 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967845678329468 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666567325592 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024593457579612732 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933911561965942 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9009638428688049 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6459866166114807 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9078232049942017 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20231688022613525 )jet/accuracy_2_of_3‚0.530340313911438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2169826328754425 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860371351242065 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968933939933777 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19591404497623444 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2371697872877121 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_4_of_4‚0.01670061983168125 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 % _timestamp‚1.7843468424122586e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.745362401008606 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6593865156173706 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5728129148483276 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9137755632400513 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48356327414512634 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.689009428024292 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.40649768710136414 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.806176483631134 epoch‚18 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943720698356628 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6379969716072083 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8220499753952026 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08159173280000687 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022501593455672264 (particle/accuracy‚0.8411961793899536 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8330498933792114 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3712516725063324 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6196552515029907 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_3_of_4‚0.07135609537363052 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6708042621612549 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.917876124382019 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1156599298119545 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3220416307449341 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23604099452495575 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7406578063964844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7293785810470581 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968922734260559 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7620211839675903 +particle/sensitivity‚0.6140433549880981 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23200365900993347 6validation_average_jet_accuracy‚0.3612845242023468 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22438795864582062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04262278601527214 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333134651184 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _runtime‚6501.106541286 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6212670803070068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02570841647684574 +particle/specificity‚0.9033209681510925 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18263176083564758 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1773572862148285 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2226872444152832 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904700398445129 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22266042232513428 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47574642300605774 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22438795864582062 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7392363548278809 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7662058472633362 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6927998065948486 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.958513617515564 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09710890799760818 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7599 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.634980320930481 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09965768456459045 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022501593455672264 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 *jet/accuracy_1_of_3‚0.9116874933242798 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21625687181949615 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8464217185974121 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6757575869560242 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.31818845868110657 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9792324304580688 &particle/f_score‚0.624159038066864 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4047613739967346 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18117578327655792 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6543004512786865 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7710614204406738 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7362521290779114 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8787442445755005 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025618042796850204 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16928745806217194 +jet/accuracy_2_of_2‚0.13858239352703094 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01839819736778736 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17056813836097717 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03521006926894188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7351605892181396 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21298401057720184 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.643523633480072 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6462274193763733 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6091539263725281 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968-£–~<ùæ{ 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.643523633480072 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6462274193763733 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6091539263725281 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518066167831421 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19079573452472687 particle/accuracy_3_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025080248713493347 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47483325004577637 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4887025058269501 particle/accuracy_4_of_5‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.776465892791748 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03490466624498367 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751248240470886 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚7.905027469348963 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8950256109237671 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_1_of_1‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5020833611488342 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2822962701320648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919931292533875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933842420578003 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8787442445755005 +jet/accuracy_3_of_3‚0.06283833086490631 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5649901628494263 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955137372016907 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6341164708137512 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7297318577766418 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6817772388458252 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6337522864341736 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2823989987373352 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3747287690639496 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996301531791687 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962449073791504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024475622922182083 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18724672496318817 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.804215669631958 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.897014319896698 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.049152832478284836 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9550725817680359 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1949562132358551 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.634980320930481 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7394860982894897 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3216533064842224 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20820482075214386 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31449222564697266 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867221117019653 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_3‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946061372756958 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_4‚0.7811702489852905 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2106267809867859 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967845678329468 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666567325592 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024593457579612732 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933911561965942 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9009638428688049 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6459866166114807 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9078232049942017 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20231688022613525 )jet/accuracy_2_of_3‚0.530340313911438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2169826328754425 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860371351242065 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968933939933777 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19591404497623444 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2371697872877121 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_4_of_4‚0.01670061983168125 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 % _timestamp‚1.7843468424122586e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.745362401008606 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6593865156173706 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5728129148483276 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9137755632400513 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48356327414512634 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.689009428024292 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.40649768710136414 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.806176483631134 epoch‚18 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943720698356628 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6379969716072083 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8220499753952026 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08159173280000687 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022501593455672264 (particle/accuracy‚0.8411961793899536 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8330498933792114 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3712516725063324 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6196552515029907 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_3_of_4‚0.07135609537363052 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6708042621612549 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.917876124382019 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1156599298119545 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3220416307449341 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23604099452495575 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7406578063964844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7293785810470581 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968922734260559 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7620211839675903 +particle/sensitivity‚0.6140433549880981 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23200365900993347 6validation_average_jet_accuracy‚0.3612845242023468 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967845678329468 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22438795864582062 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04262278601527214 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333134651184 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _runtime‚6501.106541286 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6212670803070068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02570841647684574 +particle/specificity‚0.9033209681510925 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18263176083564758 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1773572862148285 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2226872444152832 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904700398445129 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22266042232513428 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47574642300605774 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22438795864582062 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7392363548278809 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7662058472633362 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6927998065948486 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.958513617515564 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09710890799760818 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7599 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.634980320930481 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09965768456459045 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022501593455672264 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 *jet/accuracy_1_of_3‚0.9116874933242798 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21625687181949615 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8464217185974121 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6757575869560242 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.31818845868110657 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9792324304580688 &particle/f_score‚0.624159038066864 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4047613739967346 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18117578327655792 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6543004512786865 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7710614204406738 particle/accuracy_2_of_2‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7362521290779114 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8787442445755005 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025618042796850204 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16928745806217194 +jet/accuracy_2_of_2‚0.13858239352703094 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01839819736778736 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17056813836097717 :"Purity/1FRt1SRqqt0FBt/event_p`Dˆvnurity‚0.03521006926894188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7351605892181396 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21298401057720184òuΖ%žú‚(:“ ãñëÒÖؤ” gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×í'èû‚(:Ý ãñëÒ©”¤• disk.sda1.in‚0 disk.sda.out‚ 212.515625" disk.nvme0n1.in‚12757.69140625% disk.nvme0n1.out‚23067.8095703125 disk.dm-0.in‚12757.69140625 disk.sda3.out‚ 212.515625 cpu‚26.188060929159633) disk./.usagePercent‚40.24838706037158# disk./.usageGB‚599.730052947998 disk.sda1.out‚0" disk.dm-0.out‚23067.8095703125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 212.515625' disk.nvme0c0n1.out‚23067.8095703125( proc.memory.availableMB‚ 471.62890625 disk.sda2.in‚0* proc.memory.percent‚16.762900352478027 disk.drbd2165.in‚0$ memory_percent‚97.12140560150146# proc.memory.rssMB‚ 2746.43359375 proc.cpu.threads‚17 disk.rbd1.out‚ 16.01953125 network.sent‚ 1.2052042e+07 network.recv‚ 1.782273e+06 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚12757.69140625ƒ:Z1žü‚(:“ òñëÒþ˜Ý“ gpu.0.gpu‚59.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øòÒðý‚(:å òñëÒ°ÎÔ–& disk.nvme0n1.out‚23072.47607421875 disk.rbd1.out‚ 16.01953125 disk.sda2.in‚0 disk.sda3.out‚ 212.74609375$ disk.nvme0c0n1.in‚12757.69140625 disk.rbd1.in‚0 network.recv‚ 1.787231e+06$ disk./.usageGB‚599.7303314208984 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 212.74609375 disk.sda3.in‚ 0.1328125 cpu‚26.188572680726498 network.sent‚ 1.2109619e+07 disk.drbd2165.out‚0 disk.dm-0.in‚12757.69140625 disk.dm-1.in‚ 0.1328125" disk.nvme0n1.in‚12757.69140625$ memory_percent‚97.12305068969727 proc.cpu.threads‚17) disk./.usagePercent‚40.24840574892157( disk.nvme0c0n1.out‚23072.47607421875 disk.sda2.out‚0& proc.memory.availableMB‚ 471.359375# proc.memory.rssMB‚ 2746.43359375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.762900352478027 disk.dm-1.out‚ 212.74609375# disk.dm-0.out‚23072.47607421875Û”CEžþ‚(:“ òëÒÜŽË“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.748' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.349599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kGúÕ×ÿ‚(:Ì òëÒ ãç•! disk.dm-0.out‚23094.263671875 disk.sda2.in‚0 disk.sda3.out‚ 214.59375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 12758.65625* proc.memory.percent‚16.762900352478027 disk.sda.out‚ 214.59375 disk.sda2.out‚0 network.sent‚ 1.21146e+07 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 214.59375 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚23094.263671875 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2746.43359375$ memory_percent‚97.12307453155518( proc.memory.availableMB‚ 471.35546875) disk./.usagePercent‚40.24845131826267 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚23094.263671875 disk.rbd1.out‚ 16.08203125 cpu‚26.18894836794243$ disk./.usageGB‚599.7310104370117! disk.nvme0c0n1.in‚ 12758.65625 disk.dm-0.in‚ 12758.65625 disk.sda.in‚ 0.1328125 disk.rbd1.in‚0 network.recv‚ 1.790962e+06-É–ƒ2€- _wandbruntime‚6541  _runtime‚6541h› bÈ _runtime‚ 6541.04742701 _step‚627 ' lr-AdamW/pg1‚0.00031381253185674876 ' lr-AdamW/pg2‚0.00031381253185674876 trainer/global_step‚7649 % _timestamp‚1.7843468823537824e+09°Ù;,Í‚Ç ' lr-AdamW/pg1‚0.00031381253185674876 ' lr-AdamW/pg2‚0.00031381253185674876 trainer/global_step‚7649 % _timestamp‚1.7843468823537824e+09 _runtime‚ 6541.04742701 _step‚627óY¾É2ƒ- _wandbruntime‚6541  _runtime‚6541ÞΩXó'„íO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70874096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16454194 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256156292900864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8081928466817 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1171185400676864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13403008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1171185759055360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60681316 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70864119 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13403011 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70864119 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264334823574528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1157106524704768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8093933691392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1163091466985472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1163091466985472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1171185400676864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1165188453171585 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70874096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29856955 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60681314 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16453947 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8094275030528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784346882354973e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131545433 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1163091484024832 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60681314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8178625542144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8178530673664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1165188095582335 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60681316 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264334937055232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7649 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8081587916927 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1163091484024832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256156311513088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1157106507665408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1171185759055360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚628 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8094275030528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8093933691392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60681314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8178625542144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8178530673664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1165188095582335 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60681316 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264334937055232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7649 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8081587916927 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1163091484024832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256156311513088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1157106507665408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1171185759055360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚628 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8094275030528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8093933691392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1171185400676864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13403008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1171185759055360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60681316 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70864119 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13403011 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70864119 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264334823574528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1157106524704768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8093933691392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1163091466985472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1163091466985472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1171185400676864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1165188453171585 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70874096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29856955 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60681314 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16453947 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8094275030528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784346882354973e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169ôÅ'¤2†- _wandbruntime‚6541  _runtime‚6541mÿ¬ìŠ‡÷M \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1163238725054464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29860722 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚6541.661605305 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1163238725054464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1157253035376640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784346882967719e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1171334012719104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095287664640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1171333654333440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8082598970649 _step‚629 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8094946318336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8082939520651 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264369384781312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131572110 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16455969 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStaz ?s tsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1171334012719104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13404753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163238708015104 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1165335974897291 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095287664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60688946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256189740271616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8179549648384¡Tþ‰'ˆüM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264369271307776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256189740271616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8179549648384 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1163238725054464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29860722 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚6541.661605305 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1163238725054464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1157253035376640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784346882967719e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1171334012719104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095287664640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1171333654333440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8082598970649 _step‚629 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8094946318336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8082939520651 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264369384781312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131572110 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16455969 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1171334012719104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13404753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163238708015104 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1165335974897291 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095287664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60688946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71328548 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71318571 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170453535506432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1178598506038784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266057659152384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16557876 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266057762147328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30048381 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13490261 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8132544953725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1178598506038784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1178598147660288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61062916 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61062914 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132391464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1172564603252093 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170453518467072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1178598147660288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61062914 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71318571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1172564245662843 % _timestamp‚1.7843469226588914e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61062916 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1164432058298368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8144629193216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8229928325120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381485 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1170453518467072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16558120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257827749439488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71328548 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71318571 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170453535506432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1178598506038784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266057659152384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16557876 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266057762147328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30048381 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13490261 ^LDeviceStatsM}— ºüonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8132544953725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1178598506038784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1178598147660288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61062916 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61062914 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132391464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1172564603252093 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170453518467072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1178598147660288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61062914 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71318571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1172564245662843 % _timestamp‚1.7843469226588914e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61062916 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1164432058298368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8144629193216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8229928325120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381485 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1170453518467072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16558120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257827749439488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257861178198016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843469232742279e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8145641820160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408162 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1164578568970240 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1170600776536064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8145983166464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8133215457557 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8145983166464 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1178746759702528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1172711767388437 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8133556007559 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30052213 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170600776536064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266092132366336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61070546 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266092226965504 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1172712124977799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16560207 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1178746759702528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170600759496704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492003 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61070548 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61070546 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164578551930880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408162 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257861159585792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚7699 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30051963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16559960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170600759496704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61070548 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÞÔü×'˜ûM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257861178198016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843469232742279e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8145641820160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408162 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1164578568970240 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1170600776536064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8145983166464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8133215457557 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8145983166464 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1178746759702528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1172711767388437 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8133556007559 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30052213 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170600776536064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266092132366336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61070546 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266092226965504 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1172712124977799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16560207 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1178746759702528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170600759496704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492003 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61070548 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61070546 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164578551930880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408162 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257861159585792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚7699 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30051963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16559960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170600759496704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61070548 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1177815569948672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259499168753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peaŒÜÚ`%k‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8182820890743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8195666034176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30240550 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8183161440633 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚6622.245703124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217537 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30240278 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1179940395743351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13577511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1179940753332601 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61444516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16663039 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267780597287424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843469635519466e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71773023 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1186010894643712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8195324695040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8195324695040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16662770 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1186011253022208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267780496389632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171757574852608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71773023 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1177815586988032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1186011253022208 trainer/global_step‚7749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1186010894643712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8195666034176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8281409921536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8281327635968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259499187365888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171757591891968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61444514 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71783000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1177815569948672 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61444514 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1177815569948672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259499168753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8182820890743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8195666034176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30240550 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8183161440633 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚6622.245703124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217537 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30240278 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1179940395743351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13577511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1179940753332601 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61444516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16663039 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267780597287424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843469635519466e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71773023 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1186010894643712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8195324695040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8195324695040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16662770 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1186011253022208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267780496389632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171757574852608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71773023 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1177815586988032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1186011253022208 trainer/global_step‚7749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1186010894643712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8195666034176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8281409921536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8281327635968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259499187365888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171757591891968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61444514 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71783000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1177815569948672 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983ü¡·†e2¨- _wandbruntime‚6622  _runtime‚6622~N>Äý&©÷M NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71782075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8282436998656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚6622.858484334 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133244214 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71792066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1180087917468945 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1186159148300288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177962810978304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16664859 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267815053123072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13579256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16665115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71792066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267814954329600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30244371 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8183831944465 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171904085524480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8196678668288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452148 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8196678668288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8282356817408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8196337321984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30244112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1180088275058307 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8196337321984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259532616124416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13579253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843469641646614e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244214 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452146 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1177962810978304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1186159506685952 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133244214 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71792066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1180087917468945 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1186159148300288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177962810978304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16664859 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267815053123072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13579256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16665115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71792066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267814954329600 ;4DeÉÜ «­viceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30244371 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8183831944465 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171904085524480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8196678668288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452148 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8196678668288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8282356817408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8196337321984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30244112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1180088275058307 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8196337321984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259532616124416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13579253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843469641646614e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244214 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452146 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1177962810978304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1186159506685952 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1187316903413109 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1185177621430272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72237452 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚7799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1185177638469632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13664761 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8332577790464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1179083108446208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269503203082752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1193423641627136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30431896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72227475 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ö D)=#¹óO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72227475 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8233777927541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1187316903413109 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1185177621430272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72237452 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚7799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1185177638469632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13664761 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8332577790464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1179083108446208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269503203082752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1193423641627136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30431896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72227475 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_starBñ·¿¼t/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61826116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13664758 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134053589 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8246361536000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61826116 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61826114 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30431613€Ôù \2º- _wandbruntime‚6663  _runtime‚6663¨–1þ&»øM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261204035438592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8247032823808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16769191 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30435442 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1193572253669376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269537646974976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134070273 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1185324862459904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61833746 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1185324879499264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1193571895283712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13666503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1193572253669376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8333500160000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179229636157440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚6663.864909409 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080266 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247374170112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61833746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16768939 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72236527 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30435697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247374170112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261204054050816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72246518 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360°³Pf'¼ýM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261204035438592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8247032823808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16769191 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30435442 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1193572253669376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269537646974976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134070273 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1185324862459904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61833746 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1185324879499264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1193571895283712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13666503 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1193572253669376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8333500160000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179229636157440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚6663.864909409 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080266 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247374170112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61833746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16768939 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72236527 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30435697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247374170112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261204054050816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72246518 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194692695904367 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271225656999936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13752011 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62207714 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72691904 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192539689951232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134899620 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30622759 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚644 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8383612393472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72681927 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1200836388610560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13752008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDæ7p…eviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284394414449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296715698688 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62207716 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1200836388610560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297057037824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192539689951232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚2712257641891841mh÷'ÉñO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8296715698688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72691904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134889641 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30623038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚6703.942626801 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72681927 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62207716 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1186408642039808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296715698688 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62207716 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1200836388610560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297057037824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192539689951232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271225764189184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1186408659079168 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62207714 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8297057037824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16870751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1194693053493617 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1200836746989056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262842044606464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8383700970496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8284053864559 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192539672911872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194692695904367 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271225656999936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13752011 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62207714 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72691904 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192539689951232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134899620 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30622759 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚644 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8383612393472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72681927 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1200836388610560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13752008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284394414449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8297728325632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13753756 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271260107817472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134906325 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30626526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8285405468283 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16873038 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271260212902400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843470458637812e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8298069671936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13753753 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1186555169751040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62215346 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1194840575219323 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285064918281 _runtime‚6704.557561819 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1200985000652800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262875473364992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1200985000652800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186555152711680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Æc‚%1"ÌýM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1200985000652800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262875473364992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1200985000652800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186555152711680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8297728325632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13753756 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271260107817472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134906325 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30626526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8285405468283 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16873038 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271260212902400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843470458637812e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8298069671936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13753753 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1186555169751040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62215346 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1194840575219323 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285064918281 _runtime‚6704.557561819…Y¾ò2Í- _wandbruntime‚6704  _runtime‚6704û”ž… ΄ .loss/FRt2/detection_loss‚1.314994283557896 % _timestamp‚1.7843470458660655e+09 _runtime‚6704.559343961 2loss/SRqqt1/assignment_loss‚0.5031004745956623 2loss/SRqqt2/assignment_loss‚0.5056518339440921 _step‚646 /loss/FRt1/assignment_loss‚1.585489042863554 3loss/FBt1/assignment_loss‚0.0018179947881888925 3loss/FBt2/assignment_loss‚0.0025571850435872016 /loss/FRt1/detection_loss‚0.5606622982175395 1loss/SRqqt2/detection_loss‚0.6306852737278139 /loss/FBt2/detection_loss‚0.3184531759633859 epoch‚19 trainer/global_step‚7849 1loss/SRqqt1/detection_loss‚0.8723511542270944 /loss/FBt1/detection_loss‚0.5829834149044473 %loss/total_loss‚7.724568378545812 0loss/FRt2/assignment_loss‚0.8458222467125508³ —ω 0loss/FRt2/assignment_loss‚0.8458222467125508 .loss/FRt2/detection_loss‚1.314994283557896 % _timestamp‚1.7843470458660655e+09 _runtime‚6704.559343961 2loss/SRqqt1/assignment_loss‚0.5031004745956623 2loss/SRqqt2/assignment_loss‚0.5056518339440921 _step‚646 /loss/FRt1/assignment_loss‚1.585489042863554 3loss/FBt1/assignment_loss‚0.0018179947881888925 3loss/FBt2/assignment_loss‚0.0025571850435872016 /loss/FRt1/detection_loss‚0.5606622982175395 1loss/SRqqt2/detection_loss‚0.6306852737278139 /loss/FBt2/detection_loss‚0.3184531759633859 epoch‚19 trainer/global_step‚7849 1loss/SRqqt1/detection_loss‚0.8723511542270944 /loss/FBt1/detection_loss‚0.5829834149044473 %loss/total_loss‚7.724568378545812†©naôœÐ‚(:‘ ¦óëÒê„­“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.048399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎþîâÑ‚(:× ¦óëÒ‹¾‰– disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 221.1171875 disk.drbd2165.out‚0 disk.dm-0.in‚13039.19921875 disk.rbd1.in‚0# proc.memory.availableMB‚471.125 cpu‚26.187212237218755) disk./.usagePercent‚39.48892637362583 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0* proc.memory.percent‚16.762924194335938$ disk./.usageGB‚588.4135398864746$ disk.nvme0c0n1.in‚13039.19921875 disk.sda2.out‚0% disk.nvme0n1.out‚23205.3916015625 disk.rbd1.out‚ 16.39453125' disk.nvme0c0n1.out‚23205.3916015625 proc.memory.rssMB‚ 2746.4375$ memory_percent‚97.12448120117188 network.recv‚ 1.835357e+06 disk.sda.in‚ 0.1328125 disk.sda.out‚ 221.1171875" disk.nvme0n1.in‚13039.19921875 disk.sda3.out‚ 221.1171875 disk.rbd0.out‚0 network.sent‚ 1.240115e+07" disk.dm-0.out‚23205.3916015625 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0pm~4‘Ò‚(:† µóëÒº¼“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¨2êÓ‚(:ß µóëÒðµ×•* disk./.usagePercent‚39.488938661987476 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚13039.19921875% disk.nvme0n1.out‚23210.4970703125* proc.memory.percent‚16.762924194335938 cpu‚26.186651061512297 network.recv‚ 1.840737e+06 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚23210.4970703125 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 221.48046875 disk.drbd2165.out‚0 disk.dm-0.in‚13039.19921875 disk.sda3.out‚ 221.48046875 disk.rbd1.out‚ 16.45703125" disk.dm-0.out‚23210.4970703125 disk.rbd1.in‚0& proc.memory.availableMB‚ 471.359375 proc.cpu.threads‚17 disk.sda.out‚ 221.48046875$ memory_percent‚97.12305068969727 network.sent‚ 1.2464369e+07 disk.sda2.out‚0$ disk./.usageGB‚588.4137229919434$ disk.nvme0c0n1.in‚13039.19921875 proc.memory.rssMB‚ 2746.4375×€ ÷‘Ô‚(:† ÄóëÒԌƓ gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ-¶xÝÕ‚(:Ò ÄóëÒÁù“– disk.rbd1.out‚ 16.45703125 proc.cpu.threads‚17 network.sent‚ 1.2469275e+07" disk./.usageGB‚588.41357421875 disk.sda2.in‚0 disk.dm-1.out‚ 221.828125 proc.memory.rssMB‚ 2746.4375 disk.nvme0n1.in‚ 13112.15625 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚23213.85498046875 disk.sda3.out‚ 221.828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 13112.15625( proc.memory.availableMB‚ 471.35546875 cpu‚26.186159382657777 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 network.recv‚ 1.843952e+06) disk./.usagePercent‚39.48892867769364 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚23213.85498046875 disk.sda.out‚ 221.828125$ memory_percent‚97.12307453155518 disk.rbd1.in‚0* proc.memory.percent‚16.762924194335938& disk.nvme0n1.out‚23213.85498046875! disk.nvme0c0n1.in‚ 13112.15625ñ¹C÷2Ö- _wandbruntime‚6744  _runtime‚6744® .ñÉ×à ' lr-AdamW/pg1‚0.00030548724680888947 ' lr-AdamW/pg2‚0.00030548724680888947 trainer/global_step‚7899 % _timestamp‚1.7843470862137744e+09 _runtime‚6744.907317733 _step‚647¬“ØÇÎØÈ % _timestamp‚1.7843470862137744e+09 _runtime‚6744.907317733 _step‚647 ' lr-AdamW/pg1‚0.00030548724680888947 ' lr-AdamW/pg2‚0.00030548724680888947 trainer/global_step‚7899‡ÎL2Ù- _wandbruntime‚6744  _runtime‚6744¬ý»Äô'ÚîO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62589316 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208249135593984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62589314 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725693 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1199901741432832 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1199901724393472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8347752539648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193734175633408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30814279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8434745143808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202068845984875 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193734192672768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1199901724393472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73136379 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13839258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208249493972480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264513482532864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8347411200512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73146356 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62589314 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 % _timestamp‚1.7843470862145674e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208249135593984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8347752539648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16975018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272948227676672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264513501145088/‘µ.ù'ÛóO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30814279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8434745143808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202068845984875 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193734192672768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1199901724393472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73136379 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13839258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208249493972480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264513482532864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8347411200512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73146356 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62589314 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 % _timestamp‚1.7843470862145674e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208249135593984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8347752539648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16975018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272948227676672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264513501145088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62589316 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208249135593984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62589314 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725693 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1199901741432832 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1199901724393472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8347752539648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193734175633408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1200048965423104 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1202216367710469 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208397747636224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1208397389250560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8348765173760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272982676351488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13841006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13841003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8336021955191 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335681405189 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30817841 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16977098 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16976838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6745.526490102 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8435765060096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843470868325353e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73145431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62596948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8435859921408 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1193880703344640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1200048965423104 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1202216367710469 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208397747636224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1208397389250560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8348765173760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272982676351488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13841006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13841003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8336021955191 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335681405189 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30817841 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16977098 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16976838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6745.526490102 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8435765060096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843470868325353e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73145431 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62596948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8435859921408 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1193880703344640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274670904453120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17079063 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚7949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136571724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201059726266368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8398106702336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13926511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13926508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1207263775875072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62970916 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209445353654633 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31005571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8485983993856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31005828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚652 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8385286838375 # _timestamp‚1.78434712699951e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215661882577408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73600808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561745 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274671013739520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8398448041472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1207263775875072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1207263792914432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8486074668032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274670904453120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17079063 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚7949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136571724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201059726266368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8398106702336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13926511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13926508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1207263775875072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62970916 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209445353654633 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31005571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8485983993856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31005828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚652 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8385286838375 # _timestamp‚1.78434712699951e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215661882577408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73600808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561745 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274671013739520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8398448041472 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1207263775875072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3ŒHãk›2î- _wandbruntime‚6786  _runtime‚6786êÈJrý&ï÷M >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8399460675584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1201206219898880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6786.309596036 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31009410 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.784347127615844e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62978546 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62978548 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62978546 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73599883 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136578429 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62978548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136588422 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1215810494619648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266218349217792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399119329280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31009677 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209592517790977 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8399119329280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8487011332608 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201206236938240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73599883 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13928256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1207411016904704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209592875380339 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13928253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274705360550400 trainer/global_step‚7949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1207411033944064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17081421 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266218367830016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274705465635328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73609874 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588422 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31009410 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.784347127615844e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62978546 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62978548 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62978546 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73599883 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136578429 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62978548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136588422 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1215810494619648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266218349217792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399119329280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31009677 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209592517790977 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8399119329280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8487011332608 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201206236938240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73599883 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13928256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1207411016904704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209592875380339 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13928253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274705360550400 trainer/global_step‚7949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1207411033944064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17081421 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266218367830016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274705465635328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73609874 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588422 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ГY¸2ñ- _wandbruntime‚6786  _runtime‚6786DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267856376997888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137397797 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63352516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63352514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276393677428736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1214625844396032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74055260 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74055260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8537300430848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚7999 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223074629560832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17183094 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14013758 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276393566045184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1214625827356672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17182823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267856358385664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208385242820608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1214625827356672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6826.631360997 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63352514 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223074629560832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74045283 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1216821146145891 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267856376997888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137397797 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63352516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63352514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276393677428736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1214625844396032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74055260 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74055260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8537300430848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚7999 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223074629560832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17183094 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14013758 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276393566045184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1214625827356672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17182823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267856358385664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208385242820608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1214625827356672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6826.631360997 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63352514 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223074629560832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74045283 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1216821146145891 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31200396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14015506 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223222883217408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63360146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414481 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8449814831104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1214773085425664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17185154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74064326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31200660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276428014515712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74054335 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8538227371520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1214773068386304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17184893 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1223223241603072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8538315941376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚7999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208531753492480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freedƒæ‰ª‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208531770531840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169@}Ò'‚ýM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31200396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14015506 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223222883217408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63360146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414481 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8449814831104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1214773085425664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17185154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74064326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31200660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276428014515712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74054335 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8538227371520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1214773068386304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17184893 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1223223241603072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8538315941376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚7999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208531753492480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208531770531840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63360148 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8449814831104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267889787144192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63360146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4899999499320984 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.044722605496644974 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6970236301422119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961678385734558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1861686408519745 epoch‚19 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24991075694561005 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08395195007324219 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.646281361579895 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.880872905254364 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10186334699392319 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8527209162712097 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2099987417459488 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8486624956130981 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4807611107826233 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04521758854389191 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021864209324121475 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.060833342373371124 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949208498001099 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6901760101318359 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24914498627185822 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8679326176643372 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029344618320465088 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7013140916824341 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9802977442741394 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01860402338206768 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7198246717453003 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7546469569206238 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7766220569610596 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22199217975139618 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.7955533266067505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8734692931175232 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025803538039326668 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3817215859889984 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988688051700592 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18788695335388184 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2471129149198532 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9913489818572998 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3252021372318268 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9462217688560486 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9233944416046143 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.808712363243103 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05647846683859825 particle/accuracy_1_of_1‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19317466020584106 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38180336356163025 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5540517568588257 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7581509947776794 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_3‚0.5476939082145691 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )loss/val_total_loss‚7.863650147323331 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.646281361579895 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8679326176643372 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7486323118209839 particle/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7883976697921753 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_3‚0.9178541898727417 jet/accuracy_2_of_5‚1 _step‚659 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6799468398094177 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8075016736984253 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8615646362304688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6524726748466492 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.880872905254364 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2922004759311676 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2053932398557663 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.784347187199722e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888120293617249 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8840222954750061 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6379261016845703 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9413347840309143 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6539219617843628 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3436037600040436 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_4_of_5‚1 trainer/global_step‚7999 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7475477457046509 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42149728536605835 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7331874966621399 +particle/sensitivity‚0.6061843037605286 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901489019393921 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47543084621429443 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7875028252601624 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24093005061149597 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20756492018699646 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03662983700633049 +jet/accuracy_3_of_4‚0.08125406503677368 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +jet/accuracy_3_of_3‚0.06451531499624252 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3985728621482849 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025803538039326668 +jet/accuracy_1_of_1‚0.33172211050987244 *jet/accuracy_1_of_4‚0.9821165204048157 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020727241411805153 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6285571455955505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23030945658683777 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 +jet/accuracy_2_of_2‚0.14530375599861145 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12003489583730698 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21149900555610657 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2253977507352829 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960698485374451 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6465457081794739 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7692691087722778 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.016373563557863235 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2373317927122116 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023276451975107193 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6026415824890137 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6715109944343567 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.646547794342041 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.70281583070755 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982470273971558 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4617793560028076 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_2_of_3‚1 _runtime‚6845.905021073 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8634541630744934 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7482353448867798(“uS ŠÃ{ 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9802977442741394 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01860402338206768 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7198246717453003 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7546469569206238 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7766220569610596 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22199217975139618 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.7955533266067505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8734692931175232 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025803538039326668 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3817215859889984 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988688051700592 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18788695335388184 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2471129149198532 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9913489818572998 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3252021372318268 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9462217688560486 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9233944416046143 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.808712363243103 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05647846683859825 particle/accuracy_1_of_1‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19317466020584106 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38180336356163025 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5540517568588257 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7581509947776794 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_3‚0.5476939082145691 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )lC— Ëv2oss/val_total_loss‚7.863650147323331 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.646281361579895 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8679326176643372 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7486323118209839 particle/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7883976697921753 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_3‚0.9178541898727417 jet/accuracy_2_of_5‚1 _step‚659 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6799468398094177 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8075016736984253 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8615646362304688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6524726748466492 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.880872905254364 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2922004759311676 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2053932398557663 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 $ _timestamp‚1.784347187199722e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888120293617249 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8840222954750061 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6379261016845703 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9413347840309143 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6539219617843628 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3436037600040436 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_4_of_5‚1 trainer/global_step‚7999 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7475477457046509 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42149728536605835 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7331874966621399 +particle/sensitivity‚0.6061843037605286 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901489019393921 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47543084621429443 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7875028252601624 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24093005061149597 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20756492018699646 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03662983700633049 +jet/accuracy_3_of_4‚0.08125406503677368 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +jet/accuracy_3_of_3‚0.06451531499624252 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3985728621482849 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025803538039326668 +jet/accuracy_1_of_1‚0.33172211050987244 *jet/accuracy_1_of_4‚0.9821165204048157 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.020727241411805153 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6285571455955505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23030945658683777 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 +jet/accuracy_2_of_2‚0.14530375599861145 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12003489583730698 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21149900555610657 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2253977507352829 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960698485374451 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6465457081794739 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7692691087722778 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.016373563557863235 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2373317927122116 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023276451975107193 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6026415824890137 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6715109944343567 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.646547794342041 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.70281583070755 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982470273971558 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4617793560028076 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_2_of_3‚1 _runtime‚6845.905021073 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8634541630744934 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7482353448867798 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6876828670501709 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957754015922546 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7409873008728027 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8412543535232544 +particle/specificity‚0.9055439829826355 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20498554408550262 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5168278813362122 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23866784572601318 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10218177735805511 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997619092464447 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20210573077201843 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29498207569122314 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6881279945373535 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24580223858356476 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6102574467658997 7validation_average_jet_accuracy‚0.37112241983413696 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19239039719104767 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6898318529129028 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984380602836609 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7745744585990906 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7536531090736389 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7413404583930969 'particle/f_score‚0.6212218403816223 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5434624552726746 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6767305731773376 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48156270384788513 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956019520759583 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18117587268352509 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23866784572601318 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4899999499320984 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.044722605496644974 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6970236301422119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961678385734558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1861686408519745 epoch‚19 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24991075694561005 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08395195007324219 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.646281361579895 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.880872905254364 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10186334699392319 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8527209162712097 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2099987417459488 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8486624956130981 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4807611107826233 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04521758854389191 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021864209324121475 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.060833342373371124 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949208498001099 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6901760101318359 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24914498627185822 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8679326176643372 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029344618320465088 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7013140916824341“ÿ;^õ“‹‚(:ˆ ¼ôëÒÑÕ“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´+,ÜŒ‚(:Ñ ¼ôëÒ–ñë– disk.rbd1.in‚0$ memory_percent‚96.69480323791504# proc.memory.rssMB‚ 2644.59765625# disk./.usageGB‚588.468578338623 disk.drbd2165.in‚0 disk.dm-0.in‚ 13190.03125 disk.sda2.out‚0' proc.memory.availableMB‚ 541.5234375 disk.rbd0.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚24064.4384765625 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 16.83203125! disk.nvme0c0n1.in‚ 13190.03125 disk.rbd0.out‚0 disk.dm-1.out‚ 224.90625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 13190.03125 disk.sda3.out‚ 224.90625* proc.memory.percent‚16.141343116760254 proc.cpu.threads‚17 network.sent‚ 1.2683723e+07 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125" disk.dm-0.out‚24064.4384765625 disk.sda.out‚ 224.90625 disk.sda2.in‚0 cpu‚26.190608106586243 network.recv‚ 1.876765e+06* disk./.usagePercent‚39.492620050329855% disk.nvme0n1.out‚24064.4384765625 disk.sda3.in‚ 0.1328125'×ë‚(:’ Ëôëҹў• gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÇ@뎂(:à ËôëÒ·¸Æ– disk.sda.out‚ 225.2421875! disk.nvme0n1.in‚ 13260.3203125* proc.memory.percent‚16.141343116760254 cpu‚26.19026874978156# disk.dm-0.out‚24070.16650390625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0$ memory_percent‚96.69482707977295 network.recv‚ 1.882028e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 225.2421875 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 13260.3203125 disk.sda2.out‚0& disk.nvme0n1.out‚24070.16650390625 disk.sda3.out‚ 225.2421875( proc.memory.availableMB‚ 541.51953125# proc.memory.rssMB‚ 2644.59765625 network.sent‚ 1.2741312e+07$ disk./.usageGB‚588.4421043395996 disk.rbd1.out‚ 16.89453125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚24070.16650390625 disk.dm-0.in‚ 13260.3203125 disk.sda3.in‚ 0.1328125) disk./.usagePercent‚39.49084335804223Êÿ“‚(:ˆ ÚôëÒè«Ä“ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–íž6ë‚(:à ÚôëÒ—™– disk.rbd1.out‚ 16.89453125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 13260.3203125 disk.sda2.out‚0 cpu‚26.189822190370233 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚24073.28369140625* proc.memory.percent‚16.141343116760254) disk./.usagePercent‚39.49083823789154 disk.sda1.out‚0 disk.dm-1.out‚ 225.7578125 disk.sda2.in‚0& disk.nvme0n1.out‚24073.28369140625 network.sent‚ 1.2746604e+07 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 225.7578125( disk.nvme0c0n1.out‚24073.28369140625 disk.sda3.out‚ 225.7578125' proc.memory.availableMB‚ 541.5234375 network.recv‚ 1.885298e+06 disk.dm-0.in‚ 13260.3203125! disk.nvme0n1.in‚ 13260.3203125# proc.memory.rssMB‚ 2644.59765625$ disk./.usageGB‚588.4420280456543 disk.sda1.in‚0$ memory_percent‚96.69480323791504 disk.sda3.in‚ 0.1328125ýåL·2‘- _wandbruntime‚6886  _runtime‚6886ØËòÉ’Ã ' lr-AdamW/pg2‚0.00030042882916067723 trainer/global_step‚8049 % _timestamp‚1.7843472276177092e+09 _runtime‚6886.314410884 _step‚660 ' lr-AdamW/pg1‚0.00030042882916067723&É GÎ“È trainer/global_step‚8049 % _timestamp‚1.7843472276177092e+09 _runtime‚6886.314410884 _step‚660 ' lr-AdamW/pg1‚0.00030042882916067723 ' lr-AdamW/pg2‚0.00030042882916067723”a|çh2”- _wandbruntime‚6886  _runtime‚6886úàf2ô'•îO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74580034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269543727024128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63854780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504239537420 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63854780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138434816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8504580087310 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1226123679165708 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1226124036754958 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8517230911488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8517572250624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74570057 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63854782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14103528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138424837 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 trainer/global_step‚8049 % _timestamp‚1.7843472276244857e+09 _step‚661 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1232435034778624 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1223917445488640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74570057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1223917445488640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278150387857920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8606679446016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17318135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1232434676400128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17318392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31421663 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31421923 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232434676400128`ù£Ë–óO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63854780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138434816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8504580087310 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1226123679165708 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_H¦Ð°.pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1226124036754958 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8517230911488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8517572250624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74570057 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63854782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14103528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138424837 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 trainer/global_step‚8049 % _timestamp‚1.7843472276244857e+09 _step‚661 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1232435034778624 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1223917445488640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74570057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1223917445488640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278150387857920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8606679446016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17318135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1232434676400128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17318392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31421663 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31421923 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232434676400128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74580034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269543727024128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63854780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504239537420 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451514 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1224064686518272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31425737 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31425473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63862414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚662 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232583288442368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1232583288442368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1217765967339520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518243538432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14105273 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224064686518272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8518584884736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74589100 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1226271558480664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17320461 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278184848296960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1232582930056704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1224064703557632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138441521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8505591141144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63862412 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8518584884736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1217765950300160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14105276 # _timestamp‚1.78434722823685e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269577155782656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8607791307776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚8049 _runtime‚ 6886.93063639 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138451514 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138441521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8518243538432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278184947090432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74579109°q†ÿ&™ùM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74589100 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31425473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63862414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚662 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232583288442368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1232583288442368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1217765967339520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518243538432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14105273 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224064686518272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8518584884736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74589100 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1226271558480664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17320461 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278184848296960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1232582930056704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1224064703557632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138441521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8505591141144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63862412 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8518584884736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1217765950300160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14105276 # _timestamp‚1.78434722823685e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269577155782656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8607791307776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚8049 _runtime‚ 6886.93063639 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138451514 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138441521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8518243538432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278184947090432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74579109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17320200 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8505250591142 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451514 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1224064686518272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31425737 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E ›ƒ 0loss/FRt2/assignment_loss‚0.8270882501725166 -loss/FRt2/detection_loss‚1.09695290320503 1loss/SRqqt1/detection_loss‚0.8758317522002342 1loss/SRqqt2/detection_loss‚0.6041917100614397 /loss/FBt2/detection_loss‚0.3815310548501383 %loss/total_loss‚7.667418720273941 0loss/FRt1/assignment_loss‚1.5588461162512783 2loss/SRqqt1/assignment_loss‚0.5337322181787568 3loss/SRqqt2/assignment_loss‚0.41635382279640865 3loss/FBt1/assignment_loss‚0.0002476081192773001 1loss/FBt2/assignment_loss‚0.02181018649874413 _runtime‚6886.934049161 /loss/FRt1/detection_loss‚0.5276059524761038 % _timestamp‚1.7843472282407408e+09 _step‚663 /loss/FBt1/detection_loss‚0.8232271454640125 epoch‚20 trainer/global_step‚8049¢¦NKœˆ 1loss/SRqqt1/detection_loss‚0.8758317522002342 1loss/SRqqt2/detection_loss‚0.6041917100614397 /loss/FBt2/detection_loss‚0.3815310548501383 %loss/total_loss‚7.667418720273941 0loss/FRt1/assignment_loss‚1.5588461162512783 2loss/SRqqt1/assignment_loss‚0.5337322181787568 3loss/SRqqt2/assignment_loss‚0.41635382279640865 3loss/FBt1/assignment_loss‚0.0002476081192773001 1loss/FBt2/assignment_loss‚0.02181018649874413 _runtime‚6886.934049161 /loss/FRt1/detection_loss‚0.5276059524761038 % _timestamp‚1.7843472282407408e+09 _step‚663 /loss/FBt1/detection_loss‚0.8232271454640125 epoch‚20 trainer/global_step‚8049 0loss/FRt2/assignment_loss‚0.8270882501725166 -loss/FRt2/detection_loss‚1.09695290320503—ÄPMhž‚(:“ éôëÒÐÒ‰” gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CêÒ.îž‚(:ã éôëÒÀ£„–* proc.memory.percent‚16.141343116760254 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚24078.39599609375' proc.memory.availableMB‚ 541.5546875 disk.rbd0.in‚0 network.sent‚ 1.2751477e+07# proc.memory.rssMB‚ 2644.59765625 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 225.95703125 cpu‚26.189304373107145( disk.nvme0c0n1.out‚24078.39599609375 disk.dm-0.in‚ 13260.3203125$ disk./.usageGB‚588.4418258666992 disk.sda2.in‚0& disk.nvme0n1.out‚24078.39599609375# disk.nvme0c0n1.in‚ 13260.3203125 proc.cpu.threads‚17 disk.sda3.out‚ 225.95703125 disk.rbd0.out‚0) disk./.usagePercent‚39.49082466949223 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 16.95703125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 225.95703125 network.recv‚ 1.888957e+06! disk.nvme0n1.in‚ 13260.3203125 disk.rbd1.in‚0 disk.sda1.out‚0$ memory_percent‚96.69461250305176Õß4ª“Ÿ‚(:ˆ øôëÒø´Ñ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àkÄXé ‚(:Þ øôëÒÁ••– disk.sda.out‚ 226.2890625 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 226.2890625 disk.dm-1.out‚ 226.2890625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0( proc.memory.availableMB‚ 541.30859375 network.sent‚ 1.281444e+07 disk.rbd0.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 13260.3359375 disk.sda2.out‚0) disk./.usagePercent‚39.49083516580113$ disk./.usageGB‚588.4419822692871 disk.drbd2165.out‚0# memory_percent‚96.6961145401001 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125# proc.memory.rssMB‚ 2644.64453125 cpu‚26.18893043643241 network.recv‚ 1.893873e+06 disk.dm-0.in‚ 13260.3359375# disk.dm-0.out‚24082.00927734375 disk.rbd1.in‚0* proc.memory.percent‚16.141629219055176 disk.sda1.in‚0! disk.nvme0n1.in‚ 13260.3359375& disk.nvme0n1.out‚24082.00927734375 disk.rbd1.out‚ 16.95703125 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚24082.00927734375b®ú2¡- _wandbruntime‚6927  _runtime‚6927þHÉÇ¢Á trainer/global_step‚8099 % _timestamp‚1.7843472685808039e+09 _runtime‚6927.274416997 _step‚664 & lr-AdamW/pg1‚0.0002987327966329685 & lr-AdamW/pg2‚0.0002987327966329685`)hÌ£Æ _runtime‚6927.274416997 _step‚664 & lr-AdamW/pg1‚0.0002987327966329685 & lr-AdamW/pg2‚0.0002987327966329685 trainer/global_step‚8099 % _timestamp‚1.7843472685808039e+09˜û%2¤- _wandbruntime‚6927  _runtime‚6927ožŸ ï¥îO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1231279514009600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1224944990261248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1231279496970240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271215164950528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554856024328 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1239847423383552 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1239847781762048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568267752448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75034486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1239847423383552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271215146338304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_star‹âX\t/inactive_split_bytes.all.freed‚279873248817664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139260889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8658195250688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567926413312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6927.275770763 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ÕŽOù'¦óO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271215146338304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279873248817664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139260889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8658195250688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567926413312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6927.275770763 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1231279514009600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1224944990261248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1231279496970240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271215164950528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554856024328 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1239847423383552 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1239847781762048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568267752448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75034486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1239847423383552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888™uŒœo2§- _wandbruntime‚6927  _runtime‚6927&¨»û&¨õM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1233647350971810 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8659137629184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8556207628052 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8568939040256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75033561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31616962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279907712726016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64244012 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1231426755039232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1239996035425792 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8568939040256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279907809422336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14192526 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17424439 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1225091483893760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31617228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8659215713280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139277573 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1233647708561172 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225091500933120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚8099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17424702 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚ 6927.88898724 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1231426755039232 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139287566 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1231426737999872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75043552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271248575096832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64244012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1239996035425792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1239995677040128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64244014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8569280386560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555867078050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616ÍÚ²©úM T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1233647350971810 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8659137629184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8556207628052 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8568939040256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75033561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31616962 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279907712726016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64244012 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1231426755039232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1239996035425792 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8568939040256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279907809422336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14192526 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17424439 [FDeviceStatsMonitor.on_train_batch_eLýhNnd/requested_bytes.large_pool.freed‚1225091483893760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31617228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8659215713280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139277573 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1233647708561172 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225091500933120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚8099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17424702 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚ 6927.88898724 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1231426755039232 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139287566 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1231426737999872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75043552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271248575096832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64244012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1239996035425792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1239995677040128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64244014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8569280386560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555867078050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664š÷xÐ2ª- _wandbruntime‚6927  _runtime‚6927ù¤7 «ƒ %loss/total_loss‚8.213592082568587 0loss/FRt2/assignment_loss‚0.8091714613967471 2loss/SRqqt1/assignment_loss‚0.6651427278750687 epoch‚20 _runtime‚ 6927.89213406 _step‚667 2loss/SRqqt2/assignment_loss‚0.5260321677642648 2loss/FBt2/assignment_loss‚0.008168996748901317 1loss/SRqqt1/detection_loss‚1.0023107290575612 trainer/global_step‚8099 /loss/FBt1/detection_loss‚0.4539697439409792 /loss/FBt2/detection_loss‚0.5903308761341126 % _timestamp‚1.7843472691988509e+09 0loss/FRt1/assignment_loss‚1.6721844540701971 2loss/FBt1/assignment_loss‚0.010913542777322911 /loss/FRt1/detection_loss‚0.5652750664397556 /loss/FRt2/detection_loss‚1.2732899968005733 1loss/SRqqt2/detection_loss‚0.6368023195631025§S~³¬ˆ 0loss/FRt1/assignment_loss‚1.6721844540701971 2loss/FBt1/assignment_loss‚0.010913542777322911 /loss/FRt1/detection_loss‚0.5652750664397556 /loss/FRt2/detection_loss‚1.2732899968005733 1loss/SRqqt2/detection_loss‚0.6368023195631025 %loss/total_loss‚8.213592082568587 0loss/FRt2/assignment_loss‚0.8091714613967471 2loss/SRqqt1/assignment_loss‚0.6651427278750687 epoch‚20 _runtime‚ 6927.89213406 _step‚667 2loss/SRqqt2/assignment_loss‚0.5260321677642648 2loss/FBt2/assignment_loss‚0.008168996748901317 1loss/SRqqt1/detection_loss‚1.0023107290575612 trainer/global_step‚8099 /loss/FBt1/detection_loss‚0.4539697439409792 /loss/FBt2/detection_loss‚0.5903308761341126 % _timestamp‚1.7843472691988509e+09›P„Ðã­‚(:’ ‡õëÒÿÞª– gpu.0.gpu‚27.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Í•ë®‚(:à ‡õëҶϑ˜ network.sent‚ 1.2819569e+07) disk./.usagePercent‚39.49082646154496 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.188794973683148( disk.nvme0c0n1.out‚24085.93115234375 disk.rbd1.out‚ 17.01953125* proc.memory.percent‚16.141629219055176 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125# proc.memory.rssMB‚ 2644.64453125 disk.sda2.out‚0! disk.nvme0n1.in‚ 13316.9609375 disk.sda3.out‚ 226.7734375 disk.dm-1.out‚ 226.7734375 disk.sda.out‚ 226.7734375 disk.sda1.out‚0' proc.memory.availableMB‚ 541.3203125$ memory_percent‚96.69604301452637 network.recv‚ 1.897489e+06$ disk./.usageGB‚588.4418525695801 disk.dm-0.in‚ 13316.9609375# disk.dm-0.out‚24085.93115234375 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125& disk.nvme0n1.out‚24085.93115234375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 13316.9609375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0ÃvÍ5ž¯‚(:“ –õëÒܶۓ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð·bhæ°‚(:Û –õëÒ¸Úÿ• network.recv‚ 1.902496e+06 disk.sda2.in‚0! disk.nvme0n1.in‚ 13316.9609375 disk.sda3.out‚ 227.0078125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚24088.5927734375& proc.memory.availableMB‚ 540.890625# disk.nvme0c0n1.in‚ 13316.9609375" disk.dm-0.out‚24088.5927734375 disk.sda.out‚ 227.0078125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 13316.9609375 network.sent‚ 1.2882639e+07 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0) disk./.usagePercent‚39.49082415747716 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 17.01953125$ disk./.usageGB‚588.4418182373047% disk.nvme0n1.out‚24088.5927734375 disk.sda1.out‚0$ memory_percent‚96.69866561889648 disk.rbd1.in‚0 disk.dm-1.out‚ 227.0078125# proc.memory.rssMB‚ 2644.64453125* proc.memory.percent‚16.141629219055176 cpu‚26.18840567251491 proc.cpu.threads‚17´’‚(:ˆ ¥õëÒ¯èæ” gpu.0.gpu‚72.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'B¿eﲂ(:ä ¥õëÒú×ð– disk.sda3.out‚ 227.28515625( disk.nvme0c0n1.out‚24092.77001953125# disk.dm-0.out‚24092.77001953125 disk.sda2.out‚0$ disk./.usageGB‚588.4418067932129 disk.rbd1.out‚ 17.14453125 disk.sda2.in‚0 cpu‚26.18803022290806 network.recv‚ 1.905917e+06 disk.sda1.out‚0 disk.dm-1.out‚ 227.28515625 disk.drbd2165.in‚0 network.sent‚ 1.2887512e+07* disk./.usagePercent‚39.490823389454555! disk.nvme0n1.in‚ 13316.9609375 disk.sda.out‚ 227.28515625( proc.memory.availableMB‚ 541.33203125* proc.memory.percent‚16.141629219055176& disk.nvme0n1.out‚24092.77001953125 disk.dm-0.in‚ 13316.9609375$ memory_percent‚96.69597148895264# proc.memory.rssMB‚ 2644.64453125 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 13316.9609375 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 proc.cpu.threads‚17ˆJe=2³- _wandbruntime‚6968  _runtime‚6968åÎyßÉ´Ã trainer/global_step‚8149 % _timestamp‚1.7843473093427017e+09 _runtime‚6968.036277389 _step‚668 ' lr-AdamW/pg1‚0.00029703199874238893 ' lr-AdamW/pg2‚0.00029703199874238893Þ@âÎµÈ ' lr-AdamW/pg2‚0.00029703199874238893 trainer/global_step‚8149 % _timestamp‚1.7843473093427017e+09 _runtime‚6968.036277389 _step‚668 ' lr-AdamW/pg1‚0.00029703199874238893œÓÎâ2¶- _wandbruntime‚6968  _runtime‚6968ƒzüô'·îO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8709594018304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1238641548451840 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1238641565491200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1240875979326724 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096941 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64617982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75478961 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31804426 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1247260528745472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64617980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31804169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272886584264704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281596283375104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8709680498176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17526141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75478961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272886602876928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232270523854848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140106920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1247260170366976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8605813061126 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8618963254272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚6968.037189822 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64617982 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8618963254272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¨þýù'¸óO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64617980 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1240876336915974 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14278031 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1247260170366976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8605813061126 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8618963254272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚6968.037189822 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64617982 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8618963254272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8709594018304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1238641548451840 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1238641565491200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1240875979326724 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096941 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64617982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75478961 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31804426 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1247260528745472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64617980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31804169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272886584264704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281596283375104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8709680498176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17526141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75478961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272886602876928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232270523854848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140106920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚1210­%¹%1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8606824114960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1238788806520832 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1238788789481472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272920013023232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232417034526720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8606483564958 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8619634542080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123618 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚670 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75488013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8619634542080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123618 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64625612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31807952 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1232417017487360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140113625 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238788789481472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64625614 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75488013 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281630644440064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75498004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17528442 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281630741136384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31808218 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238788806520832 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1247408424023552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8619975888384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14279773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784347309959134e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1247408782409216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888xÝ'»ûM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75498004 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1238788806520832 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1238788789481472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272920013023232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232417034526720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8606483564958 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8619634542080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123618 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚670 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75488013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8619634542080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123618 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64625612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31807952 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1232417017487360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140113625 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238788789481472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64625614 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75488013 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281630644440064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75498004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17528442 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281630741136384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31808218 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238788806520832 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1247408424023552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8619975888384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14279773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784347309959134e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1247408782409216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8606824114960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1246003616972800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75933413«ÅßlÑ  @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1246003616972800 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1246003599933440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283318968041472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1239596057448448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8656429548034 % _timestamp‚1.7843473502615159e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1254672917350400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283319068939264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31995266 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140932993 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8760945850368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1248252129407232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274558040803328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64999582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17629985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚673 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1254672917350400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31995020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438„ì@¶ù'ÊóO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1254673275728896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17629742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75933413 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274558022191104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8669317416960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8669317416960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1246003599933440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140942972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8669658756096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64999582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14365278 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75943390 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1254673275728896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14365281 trainer/global_step‚8199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1248252486996482 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1239596040409088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1246003616972800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75933413 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1246003616972800 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1246003599933440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283318968041472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1239596057448448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8656429548034 % _timestamp‚1.7843473502615159e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1254672917350400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283319068939264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31995266 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140932993 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8760945850368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1248252129407232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274558040803328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64999582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17629985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚673 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1254672917350400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31995020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438¡ c-Ñ2Ë- _wandbruntime‚7009  _runtime‚7009·ð?]þ&ÌøM _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1246150858002432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1246150840963072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65007214 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274591450949632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140949677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65007212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75952456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1254821529392640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283353528058368 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1239742551080960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7009.567844136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚8199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75942465 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8670330043904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949677 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65007212 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14367023 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65007214 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1254821529392640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8762058496512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140959670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14367026 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959670 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8670330043904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1246150840963072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31998750 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274591469561856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1239742568120320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8670671390208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1248399651132826 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843473508740485e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959670 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8670330043904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1246150840963072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31998750 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274591469561856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1239742568120320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8670671390208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1248399651132826 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843473508740485e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140949677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65007212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75952456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1254821529392640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283353528058368 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1239742551080960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7009.567844136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚8199 J@DeviceStatsM”›L” onitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75942465 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8670330043904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949677 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65007212 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14367023 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¢•ú†2Î- _wandbruntime‚7009  _runtime‚7009»¶zâ σ epoch‚20 trainer/global_step‚8199 2loss/SRqqt2/assignment_loss‚0.5094258236831452 /loss/FRt1/detection_loss‚0.5542823206720681 1loss/SRqqt1/detection_loss‚1.0429665193737254 .loss/FBt2/detection_loss‚0.519707376148626 _step‚675 0loss/FRt2/assignment_loss‚0.7919668714273818 2loss/SRqqt1/assignment_loss‚0.5253628237346976 3loss/FBt1/assignment_loss‚0.0017781198312642385 3loss/FBt2/assignment_loss‚0.0010351302768619721 %loss/total_loss‚7.722068534310688 $ _timestamp‚1.784347350877501e+09 0loss/FRt1/assignment_loss‚1.5743519846999317 .loss/FRt2/detection_loss‚1.184086280626799 1loss/SRqqt2/detection_loss‚0.5861072109764888 /loss/FBt1/detection_loss‚0.4309980728596981 _runtime‚7009.570816925FeJЈ 0loss/FRt2/assignment_loss‚0.7919668714273818 2loss/SRqqt1/assignment_loss‚0.5253628237346976 3loss/FBt1/assignment_loss‚0.0017781198312642385 3loss/FBt2/assignment_loss‚0.0010351302768619721 %loss/total_loss‚7.722068534310688 $ _timestamp‚1.784347350877501e+09 0loss/FRt1/assignment_loss‚1.5743519846999317 .loss/FRt2/detection_loss‚1.184086280626799 1loss/SRqqt2/detection_loss‚0.5861072109764888 /loss/FBt1/detection_loss‚0.4309980728596981 _runtime‚7009.570816925 epoch‚20 trainer/global_step‚8199 2loss/SRqqt2/assignment_loss‚0.5094258236831452 /loss/FRt1/detection_loss‚0.5542823206720681 1loss/SRqqt1/detection_loss‚1.0429665193737254 .loss/FBt2/detection_loss‚0.519707376148626 _step‚675£Æ;aœÑ‚(:‘ áõëÒ§øé” gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.46' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.69200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦\„ÇëÒ‚(:à áõëÒ¦·‹–( proc.memory.availableMB‚ 541.04296875 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 228.67578125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.2964662e+07 disk.nvme0n1.in‚ 13412.578125 disk.rbd1.in‚0$ memory_percent‚96.69773578643799 disk.rbd0.in‚0( disk.nvme0c0n1.out‚24105.85400390625# proc.memory.rssMB‚ 2644.64453125 proc.cpu.threads‚17$ disk./.usageGB‚588.4414443969727 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 13412.578125( disk./.usagePercent‚39.4907990687388 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 228.67578125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.186081109223718 network.recv‚ 1.921182e+06 disk.sda.out‚ 228.67578125& disk.nvme0n1.out‚24105.85400390625 disk.rbd1.out‚ 17.26953125" disk.nvme0c0n1.in‚ 13412.578125 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚24105.85400390625* proc.memory.percent‚16.141629219055176³ÕJžÓ‚(:“ ðõëÒñ˜È“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÿ!œçÔ‚(:Ü ðõëÒÀõî– network.recv‚ 1.92627e+06 disk.sda.out‚ 228.8515625$ memory_percent‚96.69771194458008 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2644.64453125 proc.cpu.threads‚17& disk.nvme0n1.out‚24108.52978515625 network.sent‚ 1.3027635e+07$ disk./.usageGB‚588.4414100646973" disk.nvme0c0n1.in‚ 13412.578125( disk.nvme0c0n1.out‚24108.52978515625 disk.dm-0.in‚ 13412.578125 disk.sda3.out‚ 228.8515625 disk.rbd1.out‚ 17.26953125* disk./.usagePercent‚39.490796764670996 disk.dm-1.in‚ 0.1328125& proc.memory.availableMB‚ 541.046875 disk.sda2.out‚0 disk.nvme0n1.in‚ 13412.578125 disk.sda.in‚ 0.1328125# disk.dm-0.out‚24108.52978515625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚26.185609190309087 disk.rbd0.out‚0 disk.dm-1.out‚ 228.8515625 disk.drbd2165.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.141629219055176=ýÒ 2Õ- _wandbruntime‚7049  _runtime‚7049Éš9"ÉÖà ' lr-AdamW/pg1‚0.00029361654412795256 ' lr-AdamW/pg2‚0.00029361654412795256 trainer/global_step‚8249 % _timestamp‚1.7843473910407827e+09 _runtime‚7049.734452365 _step‚676ojDPÎ×È _step‚676 ' lr-AdamW/pg1‚0.00029361654412795256 ' lr-AdamW/pg2‚0.00029361654412795256 trainer/global_step‚8249 % _timestamp‚1.7843473910407827e+09 _runtime‚7049.734452365¤¿ žu2Ø- _wandbruntime‚7049  _runtime‚7049¦œÁô'ÙîO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32185675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65381180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1253365651415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1262086022712320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76397842 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76387865 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚7049.741420838 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1253365668454400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65381182 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1246921574002688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76387865 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8720354257920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843473910445244e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1255628637076990 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1253365668454400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276229460117504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706705485052 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262085664333824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285041702844928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8812224115200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276229478729728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14452531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1255628279487740 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8720012918784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141779024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14452528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8720012918784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32185409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17732881 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚8249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1262085664333824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769045 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1246921591042048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8720354257920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843473910445244e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1255628637076990 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1253365668454400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276229460117504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706705485052 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262085664333824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285041702844928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8812224115200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276229478729728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14452531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1255628279487740 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8720012918784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141779024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14452528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8720012918784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32185409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17732881 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚8249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1262085664333824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769045 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1246921591042048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8720354257920 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65388812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141785729 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1253512892444672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1255775801213334 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76396917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17734939 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2¾©¹º69 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8721366892032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8707716538774 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276262888876032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1255776158802696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14454276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708057088776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8721025545728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1262234276376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795722 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65388814 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚678 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1253512909484032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14454273 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1253512892444672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276262907488256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32189212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8813163278336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32189472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285076155142144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65388814 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76406908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552aÆ°x'ßýM trainer/global_step‚8249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65388812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141785729 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1253512892444672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1255775801213334 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76396917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17734939 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8721366892032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8707716538774 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276262888876032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1255776158802696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14454276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708057088776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8721025545728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1262234276376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795722 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65388814 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚678 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1253512909484032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14454273 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1253512892444672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276262907488256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32189212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8813163278336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32189472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285076155142144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65388814 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76406908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76406908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¦èái¡2à- _wandbruntime‚7050  _runtime‚7050¨j5 á‚ % _timestamp‚1.7843473916640625e+09 _step‚679 0loss/FRt1/assignment_loss‚1.6669268770999848 2loss/FBt1/assignment_loss‚2.48061314633769e-05 /loss/FRt1/detection_loss‚0.5453415504529001 1loss/SRqqt2/detection_loss‚0.5661105506440453 0loss/FRt2/assignment_loss‚0.9458194816761124 2loss/SRqqt1/assignment_loss‚0.5771395828927295 .loss/FRt2/detection_loss‚1.173153646160782 1loss/SRqqt1/detection_loss‚0.8230472164818396 /loss/FBt1/detection_loss‚0.6343688601027092 $loss/total_loss‚8.07852443357082 _runtime‚7050.357392632 epoch‚20 3loss/SRqqt2/assignment_loss‚0.47679576767129694 1loss/FBt2/assignment_loss‚0.03336301881786449 /loss/FBt2/detection_loss‚0.6364330754390927 trainer/global_step‚8249Üë¶y ⇠epoch‚20 3loss/SRqqt2/assignment_loss‚0.47679576767129694 1loss/FBt2/assignment_loss‚0.03336301881786449 /loss/FBt2/detection_loss‚0.6364330754390927 trainer/global_step‚8249 % _timestamp‚1.7843473916640625e+09 _step‚679 0loss/FRt1/assignment_loss‚1.6669268770999848 2loss/FBt1/assignment_loss‚2.48061314633769e-05 /loss/FRt1/detection_loss‚0.5453415504529001 1loss/SRqqt2/detection_loss‚0.5661105506440453 0loss/FRt2/assignment_loss‚0.9458194816761124 2loss/SRqqt1/assignment_loss‚0.5771395828927295 .loss/FRt2/detection_loss‚1.173153646160782 1loss/SRqqt1/detection_loss‚0.8230472164818396 /loss/FBt1/detection_loss‚0.6343688601027092 $loss/total_loss‚8.07852443357082 _runtime‚7050.357392632§ß»}Ñ“ã‚(:ˆ ŽöëÒøГ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸-æÖä‚(:Ë ŽöëÒ…ü„— disk.sda2.out‚0 disk.sda3.out‚ 229.71875$ disk./.usageGB‚588.4410858154297 disk.dm-1.in‚ 0.1328125% disk.nvme0c0n1.out‚24115.72265625# disk.nvme0n1.out‚24115.72265625 disk.sda3.in‚ 0.1328125( proc.memory.availableMB‚ 541.18359375# proc.memory.rssMB‚ 2644.64453125 network.recv‚ 1.934736e+06 disk.sda1.in‚0 disk.sda.out‚ 229.71875 disk.sda2.in‚0) disk./.usagePercent‚39.49077500403058 disk.rbd0.in‚0 disk.dm-1.out‚ 229.71875 disk.nvme0n1.in‚ 13472.40625* proc.memory.percent‚16.141629219055176 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.out‚24115.72265625! disk.nvme0c0n1.in‚ 13472.40625 cpu‚26.18468192583345 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 13472.40625 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 17.33203125$ memory_percent‚96.69687747955322 disk.sda1.out‚0 disk.rbd1.in‚0 network.sent‚ 1.3085007e+07«x²“å‚(:ˆ öëÒ³¶¯” gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ2Úèæ‚(:Ý öëÒ Ú±– network.sent‚ 1.3137679e+07 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.nvme0n1.in‚ 13472.40625 disk.rbd1.in‚0( proc.memory.availableMB‚ 541.08203125 network.recv‚ 1.94006e+06 disk.dm-0.in‚ 13472.40625& disk.nvme0n1.out‚24120.39599609375$ disk./.usageGB‚588.4412269592285 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 229.91796875 disk.dm-1.out‚ 229.91796875 disk.rbd1.out‚ 17.39453125* proc.memory.percent‚16.141629219055176! disk.nvme0c0n1.in‚ 13472.40625 cpu‚26.184402761839642$ memory_percent‚96.69749736785889) disk./.usagePercent‚39.49078447630935( disk.nvme0c0n1.out‚24120.39599609375# disk.dm-0.out‚24120.39599609375 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 229.91796875# proc.memory.rssMB‚ 2644.64453125Òb½ 2ç- _wandbruntime‚7090  _runtime‚7090ÒD#ÇèÁ & lr-AdamW/pg1‚0.0002919021067723633 & lr-AdamW/pg2‚0.0002919021067723633 trainer/global_step‚8299 % _timestamp‚1.7843474319927604e+09 _runtime‚7090.690469493 _step‚680!—ÌéÆ trainer/global_step‚8299 % _timestamp‚1.7843474319927604e+09 _runtime‚7090.690469493 _step‚680 & lr-AdamW/pg1‚0.0002919021067723633 & lr-AdamW/pg2‚0.0002919021067723633¨P–ñ^2ê- _wandbruntime‚7090  _runtime‚7090ïÄ?ô'ëîO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚8299 _step‚681 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1260727702896640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17835516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8757321971960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17835783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76852294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65762780 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8863381963264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1269498769695744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32375294 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7090.697665499 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1269498411317248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286764189332992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1269498411317248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142615076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1260727702896640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8771049759744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277900898043904 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1263004787157498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1269498769695744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1254247124635648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8770708420608 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254247107596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842317Ëà‚±ïìóO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76852294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65762780 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8863381963264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1269498769695744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32375294 VFDev¹±” iceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7090.697665499 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1269498411317248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286764189332992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1269498411317248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142615076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1260727702896640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8771049759744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277900898043904 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1263004787157498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1269498769695744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1254247124635648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8770708420608 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254247107596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842317 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚8299 _step‚681 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1260727702896640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17835516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8757321971960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17835783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977©{\ûV2í- _wandbruntime‚7091  _runtime‚7091èä•2ý&î÷M C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8758333025682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1269647023359488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142621781 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚8299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1254393618268160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚7091.309325296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65770412 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1260874943926272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142621781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17837851 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631774 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32379101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277934326802432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76861360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65770414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8772062393856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8758673575684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14541523 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1263151951293842 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65770414 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1269646664973824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1254393635307520 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1260874960965632 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14541526 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631774 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277934345414656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32379377 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8864327602176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76851369³´„P'ïüM \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1260874960965632 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14541526 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631774 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277934345414656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32379377 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8864327602176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76851369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8758333025682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1269647023359488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142621781 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚8299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1254393618268160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚7091.309325296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65770412 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1260874943926272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142621781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17837851 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631774 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32379101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277934326802432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76861360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65770414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8772062393856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8758673575684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14541523 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1263151951293842 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65770414 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1269646664973824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1254393635307520ª„t°2ð- _wandbruntime‚7091  _runtime‚7091‚Öy ñ… _runtime‚7091.312760572 0loss/FRt1/assignment_loss‚1.6486537738883196 3loss/FBt2/assignment_loss‚0.0015757007938725614 trainer/global_step‚8299 2loss/SRqqt1/assignment_loss‚0.6166781372512147 /loss/FRt2/detection_loss‚1.2395838183721581 _step‚683 /loss/FRt2/assignment_loss‚0.966453002031567 3loss/SRqqt2/assignment_loss‚0.40957790485976076 2loss/FBt1/assignment_loss‚0.001182705583510805 /loss/FBt1/detection_loss‚0.5600214639057716 %loss/total_loss‚7.874412504644112 /loss/FRt1/detection_loss‚0.5327245817882865 1loss/SRqqt1/detection_loss‚0.8573675218178285 1loss/SRqqt2/detection_loss‚0.6521343865694161 /loss/FBt2/detection_loss‚0.3884595077824056 epoch‚20 % _timestamp‚1.7843474326194434e+09úhû³òŠ 2loss/FBt1/assignment_loss‚0.001182705583510805 /loss/FBt1/detection_loss‚0.5600214639057716 %loss/total_loss‚7.874412504644112 /loss/FRt1/detection_loss‚0.5327245817882865 1loss/SRqqt1/detection_loss‚0.8573675218178285 1loss/SRqqt2/detection_loss‚0.6521343865694161 /loss/FBt2/detection_loss‚0.3884595077824056 epoch‚20 % _timestamp‚1.7843474326194434e+09 _runtime‚7091.312760572 0loss/FRt1/assignment_loss‚1.6486537738883196 3loss/FBt2/assignment_loss‚0.0015757007938725614 trainer/global_step‚8299 2loss/SRqqt1/assignment_loss‚0.6166781372512147 /loss/FRt2/detection_loss‚1.2395838183721581 _step‚683 /loss/FRt2/assignment_loss‚0.966453002031567 3loss/SRqqt2/assignment_loss‚0.40957790485976076«·~œ“ó‚(:ˆ ¬öëÒ°Ä” gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wø“'éô‚(:Þ ¬öëÒƬ¨— disk.sda.out‚ 230.11328125 disk.sda1.in‚0 disk.dm-1.out‚ 230.11328125 disk.drbd2165.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚24123.45556640625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 17.39453125$ memory_percent‚96.69792652130127( proc.memory.availableMB‚ 541.01171875# disk.dm-0.out‚24123.45556640625# proc.memory.rssMB‚ 2644.64453125$ disk./.usageGB‚588.4414901733398 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 13472.40625 disk.sda2.in‚0 disk.nvme0n1.in‚ 13472.40625 cpu‚26.183867017573125 network.sent‚ 1.3155563e+07 network.recv‚ 1.943479e+06) disk./.usagePercent‚39.49080214082921 disk.sda3.in‚ 0.1328125* proc.memory.percent‚16.141629219055176 proc.cpu.threads‚17 disk.drbd2165.in‚0& disk.nvme0n1.out‚24123.45556640625 disk.sda3.out‚ 230.11328125! disk.nvme0c0n1.in‚ 13472.40625ñis·“õ‚(:ˆ »öëÒõ î“ gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËDãö‚(:Ø »öëÒêÏô•* disk./.usagePercent‚39.490837213861404 disk.rbd1.in‚0 cpu‚26.183281226840393 disk.dm-0.in‚ 13575.4375 disk.rbd1.out‚ 17.45703125$ disk./.usageGB‚588.4420127868652 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.nvme0n1.in‚ 13575.4375* proc.memory.percent‚16.141629219055176 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 230.41796875 disk.nvme0c0n1.in‚ 13575.4375( proc.memory.availableMB‚ 540.82421875 network.sent‚ 1.3205533e+07 network.recv‚ 1.94873e+06" disk.dm-0.out‚24131.2724609375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 230.41796875 disk.sda3.out‚ 230.41796875# proc.memory.rssMB‚ 2644.64453125 disk.drbd2165.out‚0% disk.nvme0n1.out‚24131.2724609375$ memory_percent‚96.69907093048096 proc.cpu.threads‚17' disk.nvme0c0n1.out‚24131.2724609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.in‚0 #ª12÷‚(:“ ÊöëÒÒ¡„– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.534' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.106799999999996 gpu.0.smCloc»h]Ílk‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡öP‹×ø‚(:Ì ÊöëҨ꩗# memory_percent‚96.6975212097168 network.sent‚ 1.32105e+07 disk.nvme0c0n1.in‚ 13575.4375' disk.nvme0c0n1.out‚24138.2978515625" disk.dm-0.out‚24138.2978515625 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.952091e+06 disk.sda2.out‚0 disk.nvme0n1.in‚ 13575.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0* proc.memory.percent‚16.141629219055176$ disk./.usageGB‚588.4518280029297 disk.dm-0.in‚ 13575.4375 disk.sda2.in‚0 disk.sda3.out‚ 230.890625 disk.rbd1.out‚ 17.45703125& proc.memory.availableMB‚ 541.078125) disk./.usagePercent‚39.49149592124701 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 230.890625% disk.nvme0n1.out‚24138.2978515625 disk.dm-1.out‚ 230.890625# proc.memory.rssMB‚ 2644.64453125 cpu‚26.18306084491246 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125ÿØ„³2ù- _wandbruntime‚7131  _runtime‚7131¼’¹×Åú¿ trainer/global_step‚8349 $ _timestamp‚1.784347472911762e+09 _runtime‚ 7131.60534901 _step‚684 & lr-AdamW/pg1‚0.0002901833427763487 & lr-AdamW/pg2‚0.0002901833427763487Íž™ÞÊûÄ trainer/global_step‚8349 $ _timestamp‚1.784347472911762e+09 _runtime‚ 7131.60534901 _step‚684 & lr-AdamW/pg1‚0.0002901833427763487 & lr-AdamW/pg2‚0.0002901833427763487¬cA/l2ü- _wandbruntime‚7131  _runtime‚7131Ìx dó'ýíO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8821745261568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288486718509568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32566103 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143441149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14627031 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279572354582528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1270380937238006 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1270380579648756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66144380 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8807938458868 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66144382 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77296769 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1276911158300672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17939072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1276911158300672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1268089754378240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279572335970304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1261572658229248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1268089771417600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288486825698816 $ _timestamp‚1.784347472912558e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143451128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1261572641189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8808279008758 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66144382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66144380 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8914382539264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1276911516679168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8821745261568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1276911158300672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17939072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1276911158300672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1268089754378240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279572335970304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1261572658229248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1268089771417600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288486825698816 $ _timestamp‚1.784347472912558e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143451128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1261572641189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8808279008758 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66144382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66144380 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8914382539264 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1276911516679168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8821745261568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979­g*2ÿ- _wandbruntime‚7132  _runtime‚7132Çê²öþ&€øM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1277059411957248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605783341056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8915493254144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1268237012447232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚7132.220599851 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1268236995407872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66152012 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77315812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66152014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288521169413120 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261719168901120 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77315812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚686 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457833 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1268236995407872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261719151861760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1270528101374350 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66152014 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32569591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8822416549376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467826 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66152012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77305821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77305821 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605764728832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7843474735268707e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14628776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467826 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14628773 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1268237012447232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822416549376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17940818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32569846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277059770342912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1270528458963712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288521276595200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088ѳó˜býM XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1270528458963712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288521276595200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12770cÒw™¡$59411957248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605783341056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8915493254144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1268237012447232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚7132.220599851 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1268236995407872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66152012 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77315812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66152014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288521169413120 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261719168901120 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77315812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚686 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457833 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1268236995407872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261719151861760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1270528101374350 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66152014 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32569591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8822416549376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467826 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66152012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77305821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77305821 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605764728832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7843474735268707e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14628776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467826 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14628773 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1268237012447232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822416549376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17940818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32569846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277059770342912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0®A’£2‚- _wandbruntime‚7132  _runtime‚7132°ÏMe ƒ† _runtime‚7132.224708665 /loss/FRt1/detection_loss‚0.5452473400779508 1loss/SRqqt1/detection_loss‚0.8540218826585814 /loss/FBt1/detection_loss‚0.6953050114994718 0loss/FBt2/detection_loss‚0.47972165867292754 &loss/total_loss‚7.7597696047028535 epoch‚20 2loss/SRqqt2/assignment_loss‚0.5079976593874114 4loss/FBt1/assignment_loss‚0.00019795206000652201 % _timestamp‚1.7843474735314314e+09 _step‚687 0loss/FRt1/assignment_loss‚1.6391120045273393 0loss/FRt2/assignment_loss‚0.7866903105468461 2loss/SRqqt1/assignment_loss‚0.5249586920398067 1loss/FBt2/assignment_loss‚0.01302015414622084 .loss/FRt2/detection_loss‚1.133567378639727 1loss/SRqqt2/detection_loss‚0.5799295604465642 trainer/global_step‚8349ì-µ'„‹ _step‚687 0loss/FRt1/assignment_loss‚1.6391120045273393 0loss/FRt2/assignment_loss‚0.7866903105468461 2loss/SRqqt1/assignment_loss‚0.5249586920398067 1loss/FBt2/assignment_loss‚0.01302015414622084 .loss/FRt2/detection_loss‚1.133567378639727 1loss/SRqqt2/detection_loss‚0.5799295604465642 trainer/global_step‚8349 _runtime‚7132.224708665 /loss/FRt1/detection_loss‚0.5452473400779508 1loss/SRqqt1/detection_loss‚0.8540218826585814 /loss/FBt1/detection_loss‚0.6953050114994718 0loss/FBt2/detection_loss‚0.47972165867292754 &loss/total_loss‚7.7597696047028535 epoch‚20 2loss/SRqqt2/assignment_loss‚0.5079976593874114 4loss/FBt1/assignment_loss‚0.00019795206000652201 % _timestamp‚1.7843474735314314e+09¯Ý(´-“…‚(:ˆ ÙöëÒ̼” gpu.0.gpu‚43.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õêö䆂(:Ù ÙöëÒžë— disk.rbd1.in‚0 disk.rbd1.out‚ 17.51953125$ disk./.usageGB‚588.4526519775391 disk.dm-0.in‚ 13575.4375$ memory_percent‚96.69783115386963 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 231.10546875 disk.nvme0n1.in‚ 13575.4375 disk.sda3.out‚ 231.10546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚24145.18896484375 network.sent‚ 1.3215562e+07( disk./.usagePercent‚39.4915512188744 disk.nvme0c0n1.in‚ 13575.4375# disk.dm-0.out‚24145.18896484375 disk.sda2.in‚0# proc.memory.rssMB‚ 2644.69140625 disk.dm-1.out‚ 231.10546875 cpu‚26.18261059799134( proc.memory.availableMB‚ 541.02734375 network.recv‚ 1.955577e+06 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125* proc.memory.percent‚16.141915321350098 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚24145.18896484375ÿE ‡‚(:’ èöëÒ gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.162' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.83239999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óY!܈‚(:Ñ èöëÒåØæ• disk.rbd1.out‚ 17.51953125 disk.nvme0n1.in‚ 13575.4375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 231.23828125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 13575.4375% disk.nvme0c0n1.out‚24148.79296875# disk.nvme0n1.out‚24148.79296875 cpu‚26.182168792906435 network.sent‚ 1.3278664e+07$ disk./.usageGB‚588.4507789611816 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.960625e+06( proc.memory.availableMB‚ 540.85546875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 231.23828125 disk.dm-0.in‚ 13575.4375 disk.sda.out‚ 231.23828125 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2644.73828125) proc.memory.percent‚16.14220142364502) disk./.usagePercent‚39.49142551917509 disk.drbd2165.in‚0 disk.dm-0.out‚24148.79296875 disk.sda2.out‚0$ memory_percent‚96.69888019561768H~ Ë“‰‚(:ˆ ÷öëÒÁ”È” gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^)ä ÛŠ‚(:Ð ÷öëÒäΈ•) disk./.usagePercent‚39.49143089533331# disk.nvme0c0n1.in‚ 13638.6484375# disk.dm-0.out‚24156.72021484375* proc.memory.percent‚16.142892837524414 cpu‚26.18186423919043 disk.rbd0.in‚0 disk.rbd1.out‚ 17.58203125$ disk./.usageGB‚588.4508590698242 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚24156.72021484375 disk.dm-0.in‚ 13638.6484375 network.recv‚ 1.964053e+06 disk.sda1.out‚0 disk.sda3.out‚231.75 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.out‚231.75 disk.sda2.out‚0! disk.nvme0n1.in‚ 13638.6484375& disk.nvme0n1.out‚24156.72021484375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 540.60546875 network.sent‚ 1.3283734e+07 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚231.75 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚96.70040607452393" proc.memory.rssMB‚ 2644.8515625²Â¸û2‹- _wandbruntime‚7172  _runtime‚7172¼NÔkȌ ' lr-AdamW/pg1‚0.00028846036253290846 ' lr-AdamW/pg2‚0.00028846036253290846 trainer/global_step‚8399 % _timestamp‚1.7843475136737654e+09 _runtime‚ 7172.37046477 _step‚6881ÕÊÈÍÇ ' lr-AdamW/pg1‚0.00028846036253290846 ' lr-AdamW/pg2‚0.00028846036253290846 trainer/global_step‚8399 % _timestamp‚1.7843475136737654e+09 _runtime‚ 7172.37046477 _step‚688°.[$2Ž- _wandbruntime‚7172  _runtime‚7172`”!·õ'ïO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281243792508928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8872440763392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144287180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290209425638912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8965633129984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8858554945776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1275451822899200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1284323905284096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66525980 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32755696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872099424256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32755979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66525982 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚7172.375106926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284323905284096 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1277757087318514 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1275451822899200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77751221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77761198 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1277756729729264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14714281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18041698 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1268898191822848 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1268898174783488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77751221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8965531970048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1284324263662592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8872440763392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14714278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843475136812596e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277201 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66525980 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1275451805859840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872099424256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32755979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66525982 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚7172.375106926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284323905284096 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1277757087318514 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1275451822899200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77751221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77761198 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1277756729729264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14714281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18041698 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1268898191822848 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1268898174783488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77751221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8965531970048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMÖì8¦ onitor.on_train_batch_start/active_bytes.all.allocated‚1284324263662592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8872440763392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14714278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843475136812596e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277201 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66525980 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1275451805859840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576±¶+!2‘- _wandbruntime‚7172  _runtime‚7172¯q/Uü&’öM QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293885 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269044702494720 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1284472517326336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66533612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8859906549500 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1284472517326336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303878 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77760273 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290243751740416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873112051200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32759478 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281277202655232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18043455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66533612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8966643946496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8873453397504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8966549085184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8859565999498 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66533614 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77770264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66533614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1284472158940672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77760273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚8399ňìä'“ûM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32759750 _runtime‚7172.987686846 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77770264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 # _timestamp‚1.78434751429389e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18043724 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1277904251454858 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144293885 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32759478 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281277202655232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18043455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66533612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8966643946496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8873453397504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8966549085184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8859565999498 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66533614 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77770264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66533614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1284472158940672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77760273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚8399 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293885 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269044702494720 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1284472517326336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66533612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8859906549500 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1284472517326336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303878 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77760273 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290243751740416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873112051200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6189661622047424 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026597218587994576 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963605999946594 epoch‚20 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8806153535842896 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31614479422569275 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7577941417694092 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24401812255382538 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2367427945137024 +jet/accuracy_2_of_2‚0.14662672579288483 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7662353515625 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026133384555578232 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.691043496131897 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7447521686553955 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30547434091567993 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2551271319389343 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22115996479988098 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9411929845809937 6validation_average_jet_accuracy‚0.3940463364124298 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6595824956893921 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8474301695823669 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5740205645561218 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 *jet/accuracy_1_of_2‚0.7887963056564331 +jet/accuracy_4_of_4‚0.01628045178949833 ("ôãJ^Ÿ%Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21503546833992004 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46904605627059937 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.638809323310852 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5370833277702332 *jet/accuracy_1_of_3‚0.9184449315071106 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895139336585999 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.298096239566803 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9419354796409607 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23630090057849884 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8058342933654785 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12440356612205505 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8468360900878906 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7114659547805786 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6422774791717529 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7738385796546936 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9285714030265808 +jet/accuracy_3_of_4‚0.07758814096450806 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10646992921829224 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/specificity‚0.9093173146247864 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99079430103302 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6426801681518555 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7715680003166199 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24143216013908386 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3045869469642639 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21792881190776825 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6505579948425293 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3171361982822418 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02945738658308983 *jet/accuracy_2_of_3‚0.5503665804862976 *jet/accuracy_2_of_4‚0.7852197289466858 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.66124427318573 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23114316165447235 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8721376061439514 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 _step‚692 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7558206915855408 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28681960701942444 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23423931002616882 trainer/global_step‚8399 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3559604287147522 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30704763531684875 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028763245791196823 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99689781665802 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6256877183914185 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6826359629631042 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39115360379219055 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.480795681476593 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8474301695823669 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37991634011268616 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3123100697994232 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9642857313156128 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968403577804565 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48872193694114685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6821691393852234 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.921983540058136 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7554327845573425 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09989427030086517 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018703408539295197 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8495540618896484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972602128982544 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.746593713760376 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9790278673171997 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.344610333442688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.775434672832489 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056823331862688065 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018703408539295197 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21098829805850983 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843475330946062e+09 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.389876127243042 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7088929414749146 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.745842456817627 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885987639427185 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962538480758667 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7467360496520996 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4829496741294861 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7001727223396301 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6561839580535889 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 (particle/accuracy‚0.8469626307487488 jet/accuracy_2_of_5‚1 _runtime‚7191.789125041 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6923523545265198 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6635293960571289 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047343671321868896 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10569910705089569 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.66124427318573 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828£  Ox=›òz 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2367427945137024 +jet/accuracy_2_of_2‚0.14662672579288483 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7662353515625 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026133384555578232 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.691043496131897 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7447521686553955 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30547434091567993 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2551271319389343 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22115996479988098 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9411929845809937 6validation_average_jet_accuracy‚0.3940463364124298 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6595824956893921 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8474301695823669 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5740205645561218 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 *jet/accuracy_1_of_2‚0.7887963056564331 +jet/accuracy_4_of_4‚0.01628045178949833 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21503546833992004 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46904605627059937 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.638809323310852 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5370833277702332 *jet/accuracy_1_of_3‚0.9184449315071106 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895139336585999 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.298096239566803 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9419354796409607 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23630090057849884 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8058342933654785 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12440356612205505 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8468360900878906 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7114659547805786 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6422774791717529 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7738385796546936 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.9285714030265808 +jet/accuracy_3_of_4‚0.07758814096450806 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10646992921829224 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/specificity‚0.9093173146247864 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99079430103302 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6426801681518555 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7715680003166199 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24143216013908386 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3045869469642639 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21792881190776825 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6505579948425293 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3171361982822418 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02945738658308983 *jet/accuracy_2_of_3‚0.5503665804862976 *jet/accuracy_2_of_4‚0.7852197289466858 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.66124427318573 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23114316165447235 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8721376061439514 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 _step‚692 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7558206915855408 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28681960701942444 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23423931002616882 trainer/global_step‚8399 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3559604287147522 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30704763531684875 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028763245791196823 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99689781665802 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6256877183914185 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6826359629631042 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39115360379219055 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.480795681476593 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8474301695823669 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37991634011268616 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3123100697994232 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9642857313156128 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968403577804565 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48872193694114685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6821691393852234 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.921983540058136 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7554327845573425 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09989427030086517 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018703408539295197 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8495540618896484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972602128982544 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.746593713760376 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9790278673171997 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.344610333442688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.775434672832489 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056823331862688065 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018703408539295197 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21098829805850983 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843475330946062e+09 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.389876127243042 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7088929414749146 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.745842456817627 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885987639427185 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962538480758667 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7467360496520996 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4829496741294861 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7001727223396301 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6561839580535889 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 (particle/accuracy‚0.8469626307487488 jet/accuracy_2_of_5‚1 _runtime‚7191.789125041 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6923523545265198 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6635293960571289 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047343671321868896 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10569910705089569 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.66124427318573 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880818128585815 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7593076229095459 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42238304018974304 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *loss/val_total_loss‚7.8486795361238775 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5666502118110657 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038914043456315994 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8050146698951721 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08778558671474457 +jet/accuracy_3_of_3‚0.06123608723282814 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8721376061439514 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7015348076820374 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3615005910396576 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2530696988105774 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7003872990608215 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30704763531684875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05154425650835037 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11030669510364532 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 'particle/f_score‚0.6346369981765747 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33181625604629517 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8622071146965027 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7399430871009827 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.8125 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.90625 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6693375706672668 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47886842489242554 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9803330302238464 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2415100336074829 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3618076741695404 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6665182113647461 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8504914045333862 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8577853441238403 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6189661622047424 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026597218587994576 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963605999946594 epoch‚20 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8806153535842896 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31614479422569275 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7577941417694092 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24401812255382538´Õ‚ï÷žœ‚(:“ •÷ëÒà€·“ gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚246.405' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.281000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02¹aBÝ‚(:Ò •÷ëҢݕ disk.dm-1.in‚ 0.1328125! disk.nvme0n1.in‚ 13638.6484375 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda3.out‚232.6875 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 743.42578125 network.sent‚ 1.3355404e+07 network.recv‚ 1.979608e+06 disk.sda.out‚232.6875# memory_percent‚95.4624891281128" proc.memory.rssMB‚ 2724.7890625# disk.nvme0c0n1.in‚ 13638.6484375' disk.nvme0c0n1.out‚24413.9462890625 disk.rbd1.in‚0) proc.memory.percent‚16.63079261779785 disk.rbd0.out‚0 disk.dm-1.out‚232.6875 disk.sda2.in‚0* disk./.usagePercent‚39.491638005428506 disk.sda1.out‚0 disk.dm-0.in‚ 13638.6484375" disk.dm-0.out‚24413.9462890625% disk.nvme0n1.out‚24413.9462890625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 17.64453125 cpu‚26.18722587082587$ disk./.usageGB‚588.4539451599121 disk.sda1.in‚0¼A“ž‚(:ˆ ¤÷ëÒØØ­“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.9046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßar¬àŸ‚(:Õ ¤÷ëÒú¹ú– disk.rbd0.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2724.8046875 cpu‚26.186794415953443 network.recv‚ 1.984222e+06 disk.sda3.out‚ 232.90625# disk.nvme0c0n1.in‚ 13638.6484375 disk.sda.out‚ 232.90625 disk.rbd1.out‚ 17.64453125 disk.sda1.in‚0 disk.dm-1.out‚ 232.90625* disk./.usagePercent‚39.491582451793576$ disk./.usageGB‚588.4531173706055! disk.nvme0n1.in‚ 13638.6484375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125$ disk.nvme0n1.out‚24664.330078125 disk.dm-1.in‚ 0.1328125& disk.nvme0c0n1.out‚24664.330078125! disk.dm-0.out‚24664.330078125* proc.memory.percent‚16.630887985229492$ memory_percent‚95.46215534210205 network.sent‚ 1.3412649e+07 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.dm-0.in‚ 13638.6484375( proc.memory.availableMB‚ 743.48046875 proc.cpu.threads‚17 disk.rbd0.in‚0xÞex“ ‚(:ˆ ³÷ëÒàŒ¸• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3646 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xâïå¡‚(:Ú ³÷ëÒ¥Ô– disk.sda.in‚ 0.1328125 network.recv‚ 1.987711e+06 disk.sda2.in‚0$ memory_percent‚95.46065330505371( disk.nvme0c0n1.out‚24672.17431640625 disk.sda.out‚ 233.30078125 disk.sda3.out‚ 233.30078125" proc.memory.rssMB‚ 2724.8203125# disk.dm-0.out‚24672.17431640625* disk./.usagePercent‚39.491587059929195 disk.dm-1.out‚ 233.30078125 disk.nvme0c0n1.in‚ 13719.1875 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0& disk.nvme0n1.out‚24672.17431640625' proc.memory.availableMB‚ 743.7265625 network.sent‚ 1.3419723e+07 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 13719.1875 disk.rbd1.out‚ 17.70703125* proc.memory.percent‚16.630983352661133 cpu‚26.186376227057078 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚588.4531860351562 disk.nvme0n1.in‚ 13719.1875 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17rçÂ`2¢- _wandbruntime‚7233  _runtime‚7233¼Ž­’ɣà ' lr-AdamW/pg1‚0.00028673327670584406 ' lr-AdamW/pg2‚0.00028673327670584406 trainer/global_step‚8449 % _timestamp‚1.7843475752357585e+09 _runtime‚7233.929311386 _step‚693…÷¶¬Î¤È ' lr-AdamW/pg2‚0.00028673327670584406 trainer/global_step‚8449 % _timestamp‚1.7843475752357585e+09 _runtime‚7233.929311386 _step‚693 ' lr-AdamW/pg1‚0.00028673327670584406µ‰&¨z2¥- _wandbruntime‚7233  _runtime‚7233Ž’- ¦îO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1293684310504448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1284743423991808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78285977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278132371591168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14804048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8940528134144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.alloM™wiÇcated‚1284743441031168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145304246 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78276000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8940528134144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1293683952125952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282931142535168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67028246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145314225 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14804051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18175562 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291966074876928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927231707808 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1284743423991808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8940869473280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293683952125952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8926891157918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32979870 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291966190454784 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1287059262749086 % _timestamp‚1.7843475752368205e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78285977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0t—±ù'§óO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67028248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78276000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278132388630528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9034950953984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32979610 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293683952125952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8926891157918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32979870 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291966190454784 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1287059262749086 % _timestamp‚1.7843475752368205e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78285977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1293684310504448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1284743423991808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78285977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1278132371591168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14804048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8940528134144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1284743441031168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145304246 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78276000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8940528134144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1293683952125952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282931142535168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67028246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145314225 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14804051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18175562 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291966074876928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927231707808 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1284743423991808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8940869473280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1284890665021440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32983386 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282964552681472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67035880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8927902211640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145320930 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1284890665021440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14805796 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284890682060800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67035880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚8449 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8941882107392 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚695 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8928242761642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284890682060800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292000637017600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚7234.543713496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292000525641216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293832205782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278278899302400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78285052 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78295043 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18177850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18177593 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14805793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1278278882263040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67035878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843475758499515e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1293832564168192 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330923 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78295043,b[\íªýM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330923 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78295043 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8941882107392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32983646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1287207142064042 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78285052 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1293832564168192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1284890665021440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32983386 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282964552681472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67035880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8927902211640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145320930 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1284890665021440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14805796 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284890682060800 REDe¸Ì viceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67035880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚8449 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8941882107392 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚695 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8928242761642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284890682060800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292000637017600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚7234.543713496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292000525641216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293832205782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278278899302400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78285052 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78295043 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18177850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18177593 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14805793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1278278882263040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67035878 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843475758499515e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1293832564168192 Q_¶ ¬… 2loss/FBt2/assignment_loss‚0.006773842932770665 /loss/FBt2/detection_loss‚0.3665258306033566 epoch‚21 3loss/SRqqt2/assignment_loss‚0.49901405942873944 4loss/FBt1/assignment_loss‚0.00033542635523277554 1loss/SRqqt1/detection_loss‚0.9263872794903301 /loss/FBt1/detection_loss‚0.8121154100600896 _runtime‚ 7234.54707412 /loss/FRt1/assignment_loss‚1.577039235211038 0loss/FRt2/assignment_loss‚0.8209387893110861 2loss/SRqqt1/assignment_loss‚0.5301918659614122 /loss/FRt1/detection_loss‚0.5464485068698325 1loss/SRqqt2/detection_loss‚0.5486757507812512 trainer/global_step‚8449 /loss/FRt2/detection_loss‚1.2212673281831306 %loss/total_loss‚7.855713325188269 % _timestamp‚1.7843475758537626e+09 _step‚696ƒ–}S­Š /loss/FRt1/assignment_loss‚1.577039235211038 0loss/FRt2/assignment_loss‚0.8209387893110861 2loss/SRqqt1/assignment_loss‚0.5301918659614122 /loss/FRt1/detection_loss‚0.5464485068698325 1loss/SRqqt2/detection_loss‚0.5486757507812512 trainer/global_step‚8449 /loss/FRt2/detection_loss‚1.2212673281831306 %loss/total_loss‚7.855713325188269 % _timestamp‚1.7843475758537626e+09 _step‚696 2loss/FBt2/assignment_loss‚0.006773842932770665 /loss/FBt2/detection_loss‚0.3665258306033566 epoch‚21 3loss/SRqqt2/assignment_loss‚0.49901405942873944 4loss/FBt1/assignment_loss‚0.00033542635523277554 1loss/SRqqt1/detection_loss‚0.9263872794903301 /loss/FBt1/detection_loss‚0.8121154100600896 _runtime‚ 7234.54707412¸,s3“®‚(:ˆ Â÷ëÒ… ì” gpu.0.gpu‚72.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚196.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.2314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é¢Hͯ‚(: Â÷ëÒŽÃå–# memory_percent‚95.4620361328125 disk.sda.out‚ 233.671875 disk.sda1.in‚0 disk.dm-0.in‚ 13719.1875 disk.nvme0n1.in‚ 13719.1875 disk.rbd1.out‚ 17.70703125 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚24676.50390625! proc.memory.availableMB‚743.5 network.sent‚ 1.3422687e+07) disk./.usagePercent‚39.49158475586139 disk.rbd0.out‚0 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚24676.50390625 cpu‚26.185956454845094 network.recv‚ 1.99127e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 proc.cpu.threads‚17$ disk./.usageGB‚588.4531517028809 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 233.671875 disk.dm-1.out‚ 233.671875" proc.memory.rssMB‚ 2724.8203125* proc.memory.percent‚16.630983352661133 disk.sda2.out‚0 disk.sda.in‚ 0.1328125# disk.nvme0n1.out‚24676.50390625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 13719.1875–VW“°‚(:ˆ Ñ÷ëÒôúê“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V„Ö±‚(:Ë Ñ÷ëÒÁ¹Ü• disk.rbd1.out‚ 17.76953125$ memory_percent‚95.46205997467041" proc.memory.rssMB‚ 2724.8203125 proc.cpu.threads‚17 network.sent‚ 1.3485838e+07& disk.nvme0n1.out‚24682.72705078125 disk./.usageGB‚ 588.453125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚24682.72705078125# disk.dm-0.out‚24682.72705078125 disk.sda3.out‚ 234.09375( proc.memory.availableMB‚ 743.49609375 network.recv‚ 1.997236e+06 disk.dm-1.out‚ 234.09375 disk.nvme0c0n1.in‚ 13719.1875 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 13719.1875 disk.dm-1.in‚ 0.1328125 disk.rbd1.in‚0* proc.memory.percent‚16.630983352661133 disk.rbd0.out‚0 disk.sda.out‚ 234.09375* disk./.usagePercent‚39.491582963808646 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚26.185444589645375 disk.sda1.in‚0 disk.dm-0.in‚ 13719.1875 disk.sda2.in‚0 disk.sda2.out‚0€ÉÈñ2²- _wandbruntime‚7274  _runtime‚7274£ RdzÁ trainer/global_step‚8499 % _timestamp‚1.7843476161517906e+09 _runtime‚7274.845408607 _step‚697 & lr-AdamW/pg1‚0.0002850021962226508 & lr-AdamW/pg2‚0.0002850021962226508ÎMò?Ì´Æ _step‚697 & lr-AdamW/pg1‚0.0002850021962226508 & lr-AdamW/pg2‚0.0002850021962226508 trainer/global_step‚8499 % _timestamp‚1.7843476161517906e+09 _runtime‚7274.845408607¹{¢ë2µ- _wandbruntime‚7274  _runtime‚7274¬Qhó'¶íO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18278586 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚293688944639488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146150277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14891298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67409848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1301097057487872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8991564975104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8991223635968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146140298 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1285457905184768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚698 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8991223635968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67409846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294435412829594 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1285457922224128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8977507644826 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1292105475473408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18278841 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33169884 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284602580461568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293688841644544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9086279795200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1292105492512768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1292105475473408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚7274.846718249 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1301096699109376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33170142 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78730452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14891301 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1301097057487872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78740429 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67409848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78740429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9086364177920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146150277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14891298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67409848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1301097057487872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8991564975104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8991223635968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146140298 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1285457905184768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚698 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8991223635968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67409846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294435412829594 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1285457922224128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8977507644826 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1292105475473408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18278841 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33169884 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284602580461568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293688841644544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9086279795200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1292105492512768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1292105475473408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚7274.846718249 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1301096699109376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33170142 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78730452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14891301 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1301097057487872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78740429 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67409848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78740429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9086364177920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156982 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1292252733542400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8992577609216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1301245311151616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1292252733542400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166975 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293723292908544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284636009220096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284635990607872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14893046 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9087388773376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9087302300672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301244952765952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33173690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18280647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8978518698548 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67417478 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33173950 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1292252716503040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8992236262912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992577609216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146156982 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1301245311151616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1294582934555188 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78749495 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8992236262912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14893043 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146166975ùýCÎ'¼ýM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67417478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1292252716503040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚7275.459766037 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67417480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293723397993472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843476167659664e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284635990607872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14893046 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9087388773376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9087302300672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301244952765952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33173690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18280647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8978518698548 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67417478 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33173950 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1292252716503040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8992236262912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992577609216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146156982 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1301245311151616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1294582934555188 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78749495 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8992236262912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14893043 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146166975 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156982 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1292252733542400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8992577609216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1301245311151616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1292252733542400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166975 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293723292908544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284636009220096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295411692978688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9137674590720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67791448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286274018387968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1301811562910102 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79194881 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1308509446092800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9028124131734 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67791446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1301811920499352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1299467526955008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9041919137792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33360945 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67791446 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79184904 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1308509446092800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79184904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299467526955008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295411692978688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9137674590720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67791448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286274018387968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1301811562910102 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79194881 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1308509446092800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9028124131734 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67791446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1301811920499352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1299467526955008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9041919137792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33360945 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67791446 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79184904 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1308509446092800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79184904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299467526955008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33364521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18384475 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67799078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147003027 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9029135185456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18384228 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286307447146496 trainer/global_step‚8549 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33364771 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1301959084635696 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146993034 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67799080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9042931764736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295446150805504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9043273111040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9043273111040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14980293 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9138703659008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9029475735458 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1299614767984640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146993034 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1301959442225058 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1299614767984640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14980296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67799080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79193956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295446049914880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79203947 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9138621380608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79203947 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843476577352326e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67799078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1308657699749376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9042931764736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295446150805504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9043273111040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9043273111040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14980293 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9138703659008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9029475735458 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1299614767984640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146993034 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1301959442225058 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1299614767984640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14980296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67799080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79193956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295446049914880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79203947 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9138621380608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79203947 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843476577352326e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67799078 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1308657699749376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33364521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18384475 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67799078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147003027 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9029135185456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003027 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18384228 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286307447146496 trainer/global_step‚8549¿ƒÇÕ¥2Í- _wandbruntime‚7316  _runtime‚7316pË1ºÎÿ /loss/FRt1/assignment_loss‚1.581803979263042 2loss/SRqqt1/assignment_loss‚0.4731107822723905 /loss/FRt2/detection_loss‚1.2528684637757317 0loss/SRqqt1/detection_loss‚0.840446451225385 1loss/SRqqt2/detection_loss‚0.6187420632987135 %loss/total_loss‚7.743420580076197 trainer/global_step‚8549 % _timestamp‚1.7843476577395666e+09 -loss/FRt2/assignment_loss‚0.8262732412651 1loss/SRqqt2/assignment_loss‚0.463945560225366 _runtime‚7316.432842154 _step‚704 3loss/FBt1/assignment_loss‚0.0013090803164619302 1loss/FBt2/assignment_loss‚0.03609100755110582 /loss/FRt1/detection_loss‚0.5519550258319237 /loss/FBt1/detection_loss‚0.62525â„K02539816002 epoch‚21 0loss/FBt2/detection_loss‚0.47162467106937656îÊ=T Ï„ 0loss/FBt2/detection_loss‚0.47162467106937656 /loss/FRt1/assignment_loss‚1.581803979263042 2loss/SRqqt1/assignment_loss‚0.4731107822723905 /loss/FRt2/detection_loss‚1.2528684637757317 0loss/SRqqt1/detection_loss‚0.840446451225385 1loss/SRqqt2/detection_loss‚0.6187420632987135 %loss/total_loss‚7.743420580076197 trainer/global_step‚8549 % _timestamp‚1.7843476577395666e+09 -loss/FRt2/assignment_loss‚0.8262732412651 1loss/SRqqt2/assignment_loss‚0.463945560225366 _runtime‚7316.432842154 _step‚704 3loss/FBt1/assignment_loss‚0.0013090803164619302 1loss/FBt2/assignment_loss‚0.03609100755110582 /loss/FRt1/detection_loss‚0.5519550258319237 /loss/FBt1/detection_loss‚0.6252502539816002 epoch‚21À-;œÐ‚(:‘ øëÒ¨—” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.093' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.818600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÔù:äÑ‚(:Ù øëÒûåú• disk.sda3.out‚ 235.40625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.out‚ 17.95703125 disk.sda1.out‚0 cpu‚26.183826687631672* disk./.usagePercent‚39.491479792772346# disk./.usageGB‚588.451587677002 disk.dm-0.in‚13794.19921875 disk.sda.out‚ 235.40625 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚95.46470642089844 disk.rbd0.in‚0" proc.memory.rssMB‚ 2724.8984375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0& disk.nvme0n1.out‚24698.49267578125* proc.memory.percent‚16.631460189819336 network.sent‚ 1.3603741e+07 disk.dm-1.in‚ 0.1328125$ proc.memory.availableMB‚743.0625 disk.dm-1.out‚ 235.40625 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚13794.19921875( disk.nvme0c0n1.out‚24698.49267578125" disk.nvme0n1.in‚13794.19921875 disk.sda3.in‚ 0.1328125 network.recv‚ 2.019627e+06 disk.rbd0.out‚0 disk.sda1.in‚0# disk.dm-0.out‚24698.49267578125‚‚F“Ò‚(:ˆ œøëÒçü“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ|¨pëÓ‚(:à œøëÒ«‘–* disk./.usagePercent‚39.491480304787416 disk.sda1.in‚0& disk.nvme0c0n1.out‚24701.349609375 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 17.95703125( proc.memory.availableMB‚ 743.30859375 disk.sda3.out‚ 235.80859375 network.sent‚ 1.3666604e+07 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚588.4515953063965 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚13794.19921875" proc.memory.rssMB‚ 2724.9453125$ disk.nvme0n1.out‚24701.349609375* proc.memory.percent‚16.631746292114258 disk.sda1.out‚0 disk.dm-1.out‚ 235.80859375 disk.drbd2165.out‚0! disk.dm-0.out‚24701.349609375" disk.nvme0n1.in‚13794.19921875 network.recv‚ 2.024036e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 235.80859375 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚26.183315109272016$ disk.nvme0c0n1.in‚13794.19921875 disk.sda2.out‚0# memory_percent‚95.4632043838501 proc.cpu.threads‚17&²× “Ô‚(:ˆ «øëÒÉÉ–” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.501' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R&³ãÕ‚(:Ø «øëÒ䑱– disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚26.182872836827755 disk.sda.in‚ 0.1328125 disk.sda2.in‚0( proc.memory.availableMB‚ 743.30859375 disk.sda1.out‚0 network.sent‚ 1.367156e+07$ disk./.usageGB‚588.4516143798828 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚16.631746292114258& disk.nvme0n1.out‚24705.67431640625 disk.sda3.out‚ 236.2421875 disk.rbd1.out‚ 18.01953125 disk.rbd0.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 13883.03125# memory_percent‚95.4632043838501" proc.memory.rssMB‚ 2724.9453125 proc.cpu.threads‚17 disk.sda.out‚ 236.2421875 network.recv‚ 2.026859e+06) disk./.usagePercent‚39.49148158482508 disk.nvme0n1.in‚ 13883.03125 disk.dm-1.out‚ 236.2421875( disk.nvme0c0n1.out‚24705.67431640625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 13883.03125# disk.dm-0.out‚24705.67431640625 disk.sda1.in‚0+À 2Ö- _wandbruntime‚7356  _runtime‚7356µ}”È× ' lr-AdamW/pg1‚0.00028152849627356415 ' lr-AdamW/pg2‚0.00028152849627356415 trainer/global_step‚8599 $ _timestamp‚1.784347697896689e+09 _runtime‚7356.590143951 _step‚705CävÍØÇ ' lr-AdamW/pg1‚0.00028152849627356415 ' lr-AdamW/pg2‚0.00028152849627356415 trainer/global_step‚8599 $ _timestamp‚1.784347697896689e+09 _runtime‚7356.590143951 _step‚705ÁÎl 2Ù- _wandbruntime‚7356  _runtime‚7356 ß“áô'ÚîO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1315922551454720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1306829578436608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9092614639616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9188698253824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18485573 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1309187712990610 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9092955978752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68173046 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315922551454720 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315922193076224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9092955978752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147822381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843476978974442e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300108989411328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9078740618642 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1306829595475968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9092614639616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚7356.591072694 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68173046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297134135955968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315922193076224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15065801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287945437702144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79649333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9092955978752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68173046 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315922551454720 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315922193076224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9092955978752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147822381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843476978974442e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300108989411328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9078740618642 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1306829595475968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9092614639616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚7356.591072694 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68173046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297134135955968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315922193076224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15065801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287945437702144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79649333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79648408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1306976819466240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9080092222366 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚8599 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297168586359808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287978885072896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1300255500083200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297168695639040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1316070805118464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33555227 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9189810566144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843476985160286e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839079 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147829086 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18487954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1300255483043840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147839079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9093627266560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚707 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1306976836505600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68180680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9093627266560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚7357.209809684 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1306976836505600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15067543Cýª'ÞýM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839079 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147829086 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18487954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1300255483043840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147839079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9093627266560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚707 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1306976836505600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68180680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9093627266560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚7357.209809684 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1306976836505600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15067543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1306976819466240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68180680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15067546 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147829086 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79648408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9079751672364 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79658399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1309335234716204 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9093968612864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79648408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1306976819466240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9080092222366 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚8599 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297168586359808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287978885072896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1300255500083200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297168695639040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1316070805118464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33555227 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9189810566144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843476985160286e+09 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1323334940059648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1323334940059648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15153048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9239822958592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9143310141440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚289616875628544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9129697655440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148648454 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1323335298438144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚7397.673714309 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80093808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9143310141440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80103785 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298856698587136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚710 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1314191646957568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80103785 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843477389799304e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148658433 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33741855 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1316564220660368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚7397.673714309 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80093808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9143310141440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80103785 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298856698587136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚7­»QçJ10 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1314191646957568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80103785 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843477389799304e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148658433 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33741855 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1316564220660368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1323334940059648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1323334940059648ÆRë)2î- _wandbruntime‚7398  _runtime‚7398¾j9×ÿ&ïùM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68562278 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33745668 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7398.290655873 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1314338887987200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18590875 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1314338887987200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843477395968666e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9144664114688 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1307581033676800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚8649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148665138 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316711742386074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18591136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9240834290176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148675131 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33745932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15154793 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298891138677248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1323483552101888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9144664114688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80102860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289650322999296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80102860 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1314338870947840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68562280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68562278 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1307581016637440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1323483193716224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298891260539392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9130708709274 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591K‘˜ø'ðþM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33745932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15154793 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298891138677248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1323483552101888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9144664114688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80102860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289650322999296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80102860 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1314338870947840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68562280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68562278 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1307581016637440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1323483193716224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298891260539392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9130708709274 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68562278 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33745668 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7398.290655873 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1314338887987200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18590875 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1314338887987200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843477395968666e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9144664114688 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1307581033676800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚8649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148665138 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316711742386074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18591136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9240834290176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148675131 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993ÇÊ›52ñ- _wandbruntime‚7398  _runtime‚7398Ë$ ò† /loss/FRt2/detection_loss‚1.2067449460025914 1loss/SRqqt2/detection_loss‚0.6012229171155193 epoch‚21 trainer/global_step‚8649 _runtime‚7398.293191061 3loss/SRqqt2/assignment_loss‚0.45011429172367595 /loss/FRt1/detection_loss‚0.5505634881691328 1loss/SRqqt1/detection_loss‚0.9929676928114912 /loss/FBt1/detection_loss‚0.6979359234805758 $ _timestamp‚1.784347739599898e+09 _step‚712 0loss/FRt1/assignment_loss‚1.5275885569421868 0loss/FRt2/assignment_loss‚0.8722421165372505 3loss/FBt2/assignment_loss‚0.0035559593528138042 %loss/total_loss‚7.837473533536522 2loss/SRqqt1/assignment_loss‚0.5111276517574157 2loss/FBt1/assignment_loss‚0.015136936501368842 0loss/FBt2/detection_loss‚0.40827305314249973¡ÿìó‹ 0loss/FRt1/assignment_loss‚1.5275885569421868 0loss/FRt2/assignment_loss‚0.8722421165372505 3loss/FBt2/assignment_loss‚0.0035559593528138042 %loss/total_loss‚7.837473533536522 2loss/SRqqt1/assignment_loss‚0.5111276517574157 2loss/FBt1/assignment_loss‚0.015136936501368842 0loss/FBt2/detection_loss‚0.40827305314249973 /loss/FRt2/detection_loss‚1.2067449460025914 1loss/SRqqt2/detection_loss‚0.6012229171155193 epoch‚21 trainer/global_step‚8649 _runtime‚7398.293191061 3loss/SRqqt2/assignment_loss‚0.45011429172367595 /loss/FRt1/detection_loss‚0.5505634881691328 1loss/SRqqt1/detection_loss‚0.9929676928114912 /loss/FBt1/detection_loss‚0.6979359234805758 $ _timestamp‚1.784347739599898e+09 _step‚712ÈŽ¶*õô‚(:’ çøëÒÇâÀ” gpu.0.gpu‚51.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.015' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.20299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙôzÆàõ‚(:Õ çøëÒ½°…—" proc.memory.rssMB‚ 2724.9921875 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 237.265625 disk.drbd2165.out‚0 disk.sda.out‚ 237.265625* disk./.usagePercent‚39.491019491225785# disk.nvme0c0n1.in‚ 13993.5390625 disk.dm-0.in‚ 13993.6796875$ disk.nvme0n1.out‚24725.201171875 disk.rbd1.in‚0 cpu‚26.18085536863005 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.dm-0.out‚24725.201171875) proc.memory.percent‚16.63203239440918 disk.sda1.in‚0" disk.nvme0n1.in‚13993.56640625" memory_percent‚96.968674659729 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 237.265625 disk.rbd1.out‚ 18.14453125 network.sent‚ 1.3748613e+07 network.recv‚ 2.041586e+06$ disk./.usageGB‚588.4447288513184 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚24725.201171875 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 496.65234375èÞ*½žö‚(:“ öøëÒñ…Ä“ gpu.0.gpu‚42.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.479200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÐç Ú÷‚(:Ï öøëÒ¬êÜ– network.recv‚ 2.045904e+06) disk./.usagePercent‚39.49102845148948 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 13995.125 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 13995.125) proc.memory.percent‚16.63203239440918 disk.sda3.out‚ 237.44140625 proc.cpu.threads‚17$ disk./.usageGB‚588.4448623657227' proc.memory.availableMB‚ 496.6484375 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 disk.drbd2165.out‚0 cpu‚26.180493699527993 network.sent‚ 1.3811609e+07 disk.sda.out‚ 237.44140625 disk.dm-1.in‚ 0.1328125$ disk.nvme0n1.out‚24729.611328125 disk.sda1.in‚0 disk.dm-1.out‚ 237.44140625$ memory_percent‚96.96869850158691" proc.memory.rssMB‚ 2724.9921875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚24729.611328125 disk.dm-0.in‚ 13995.125! disk.dm-0.out‚24729.611328125 disk.rbd1.out‚ 18.14453125ƒë@2ø- _wandbruntime‚7438  _runtime‚7438\üìÉùà ' lr-AdamW/pg1‚0.00027804015510834524 ' lr-AdamW/pg2‚0.00027804015510834524 trainer/global_step‚8699 % _timestamp‚1.7843477797737608e+09 _runtime‚7438.470467341 _step‚713­Z_uÎúÈ _runtime‚7438.470467341 _step‚713 ' lr-AdamW/pg1‚0.00027804015510834524 ' lr-AdamW/pg2‚0.00027804015510834524 trainer/global_step‚8699 % _timestamp‚1.7843477797737608e+09É÷fr 2û- _wandbruntime‚7438  _runtime‚7438þ©r¸ÒüîO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inËÕø_"'active_split_bytes.small_pool.peak‚ 153393664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33932241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1321553698439168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18691943 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1330747687043072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494485 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1323940013151626 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80558237 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚8699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9194346982400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149484506 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1330748045421568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18692208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291288332167168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68936246 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9179973592458 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68936246 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1330748045421568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300579242350080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291288313554944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1321553681399808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1314760056598528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7438.474448365 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80548260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9290813217280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843477797777479e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68936248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33932509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300579126772224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1330748045421568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18692208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291288332167168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68936246 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9179973592458 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68936246 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1330748045421568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300579242350080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291288313554944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1321553681399808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1314760056598528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7438.474448365 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80548260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9290813217280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843477797777479e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68936248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33932509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300579126772224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1330747687043072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494485 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1323940013151626 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80558237 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890Ê»ô¹Ç2þ- _wandbruntime‚7439  _runtime‚7439«!vþ&ÿøM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80557312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291321742313472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300613694295040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8699 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1330895940699648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321700922429440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18693985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9291933369344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚7439.087630517 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18694247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68943878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9180984646180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291321760925696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68943880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80557312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511183 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511183 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80567303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9291836410880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843477803937607e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80567303 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321700922429440 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33936028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9195359616512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1324087534877220 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501190 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1330896299085312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15242046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68943878 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1314906567270400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1321700939468800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1314906550231040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9181325196182 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¥cΠ’ €ýM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80557312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291321742313472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300613694295040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8699 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1330895940699648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321700922429440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18693985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9291933369344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚7439.087630517 RBDeviceStatsMonitor.on_train_batch_end/active"‹Aq_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18694247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68943878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9180984646180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291321760925696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68943880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80557312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511183 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511183 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80567303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9291836410880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843477803937607e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80567303 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321700922429440 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33936028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9195359616512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1324087534877220 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501190 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1330896299085312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15242046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68943878 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1314906567270400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1321700939468800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1314906550231040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9181325196182 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438Ës­X<2- _wandbruntime‚7439  _runtime‚7439B!ǽ ‚„ 2loss/FBt2/assignment_loss‚0.002998671203137147 1loss/SRqqt1/detection_loss‚0.7285521703974831 1loss/SRqqt2/detection_loss‚0.5815410291383283 epoch‚21 trainer/global_step‚8699 _runtime‚7439.089368367 2loss/SRqqt1/assignment_loss‚0.5793677310221056 2loss/SRqqt2/assignment_loss‚0.5085991416294894 3loss/FBt1/assignment_loss‚0.0024554815504520674 /loss/FRt1/detection_loss‚0.5648804242990727 /loss/FRt2/detection_loss‚1.2492021739768184 /loss/FBt1/detection_loss‚0.7063632021639862 /loss/FBt2/detection_loss‚0.5180617608063927 % _timestamp‚1.7843477803960707e+09 /loss/FRt2/assignment_loss‚0.735035057113819 %loss/total_loss‚7.810895960123412 0loss/FRt1/assignment_loss‚1.6338391168223274 _step‚716èá̓‰ %loss/total_loss‚7.810895960123412 0loss/FRt1/assignment_loss‚1.6338391168223274 _step‚716 2loss/FBt2/assignment_loss‚0.002998671203137147 1loss/SRqqt1/detection_loss‚0.7285521703974831 1loss/SRqqt2/detection_loss‚0.5815410291383283 epoch‚21 trainer/global_step‚8699 _runtime‚7439.089368367 2loss/SRqqt1/assignment_loss‚0.5793677310221056 2loss/SRqqt2/assignment_loss‚0.5085991416294894 3loss/FBt1/assignment_loss‚0.0024554815504520674 /loss/FRt1/detection_loss‚0.5648804242990727 /loss/FRt2/detection_loss‚1.2492021739768184 /loss/FBt1/detection_loss‚0.7063632021639862 /loss/FBt2/detection_loss‚0.5180617608063927 % _timestamp‚1.7843477803960707e+09 /loss/FRt2/assignment_loss‚0.735035057113819ÌlIÀ“„‚(:ˆ …ùëÒË´º“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.853' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ê¨Û…‚(:Ð …ùëÒ»Ïë•" disk.dm-0.out‚24736.0205078125 disk.rbd1.out‚ 18.26953125 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 13995.125 disk.sda.out‚ 237.69921875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125# memory_percent‚96.9689130783081 network.sent‚ 1.381662e+07$ disk./.usageGB‚588.4447631835938 disk.sda3.in‚ 0.1328125 cpu‚26.18010412327943 disk.dm-0.in‚ 13995.125) proc.memory.percent‚16.63203239440918) disk./.usagePercent‚39.49102179529359 disk.sda.in‚ 0.1328125 network.recv‚ 2.049467e+06% disk.nvme0n1.out‚24736.0205078125 disk.dm-1.out‚ 237.69921875 disk.sda3.out‚ 237.69921875 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚24736.0205078125 disk.nvme0n1.in‚ 13995.125" proc.memory.rssMB‚ 2724.9921875( proc.memory.availableMB‚ 496.61328125hm&Ç“†‚(:ˆ ”ùëÒ펌• gpu.0.gpu‚68.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h?þ±á‡‚(:Ö ”ùëÒÅï¿– disk.dm-0.in‚ 13995.125 disk.dm-1.out‚ 238.12890625 disk.nvme0c0n1.in‚ 13995.125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚96.96853160858154" proc.memory.rssMB‚ 2724.9921875 disk.nvme0n1.in‚ 13995.125 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 238.12890625 disk.rbd1.in‚0( proc.memory.availableMB‚ 496.67578125* disk./.usagePercent‚39.491037411753176# disk./.usageGB‚588.444995880127 disk.sda.out‚ 238.12890625& disk.nvme0n1.out‚24739.07275390625) proc.memory.percent‚16.63203239440918 network.sent‚ 1.3879617e+07 network.recv‚ 2.053953e+06( disk.nvme0c0n1.out‚24739.07275390625 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.179638106690547 disk.drbd2165.in‚0# disk.dm-0.out‚24739.07275390625 disk.rbd1.out‚ 18.26953125 proc.cpu.threads‚17 disk.drbd2165.out‚0ê8:žˆ‚(:“ £ùëÒßÛ¤• gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.444799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÂK׉‚(:Ì £ùëÒ·Åʘ) proc.memory.percent‚16.63203239440918 disk.sda1.in‚0$ memory_percent‚96.96850776672363$ disk./.usageGB‚588.4449615478516 disk.sda3.out‚ 238.5703125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 13995.125 disk.sda1.out‚0 cpu‚26.179130313043842 disk.sda.out‚ 238.5703125 disk.sda2.in‚0" proc.memory.rssMB‚ 2724.9921875 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.33203125) disk./.usagePercent‚39.49103510768537 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 13995.125 disk.sda3.in‚ 0.1328125 network.recv‚ 2.057362e+06 disk.nvme0c0n1.in‚ 13995.125! disk.dm-0.out‚24745.498046875 disk.sda2.out‚0$ disk.nvme0n1.out‚24745.498046875 disk.rbd0.in‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 496.6796875 network.sent‚ 1.3884464e+07 disk.dm-1.out‚ 238.5703125& disk.nvme0c0n1.out‚24745.498046875¦v³¡2Š- _wandbruntime‚7479  _runtime‚74798hÖ`Ç‹Á % _timestamp‚1.7843478207387187e+09 _runtime‚7479.432313081 _step‚717 & lr-AdamW/pg1‚0.0002762907739865899 & lr-AdamW/pg2‚0.0002762907739865899 trainer/global_step‚8749ܨDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69317848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18795208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9245042484224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9244701145088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9244701145088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚7479.433571437 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1322085590192128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9230930629256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81002712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1328915749920768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150320558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1338160792404992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331316163232134 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230590079366 _step‚718 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1338160434026496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81012689 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚8749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1338160792404992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327551 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81002712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1328915749920768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69317848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1338160434026496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9341946507264v®‘!ôO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9244701145088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9244701145088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚7479.433571437 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1322085590192128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9230930629256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81002712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1328915749920768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150320558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1338160792404992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331316163232134 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230590079366 _step‚718 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1338160434026496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81012689 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚8749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1338160792404992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327551 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81002712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1328915749920768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69317848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1338160434026496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bŽæ  iytes.small_pool.allocated‚ 9341946507264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34123026 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1328915732881408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69317848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18795208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9245042484224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69325478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150337242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9246055118336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69325478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292993180239872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347235 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9246055118336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302336160845312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚8749 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1329062990950400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9245713772032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1331463684957728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81011764 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347235 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34126524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81011764 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34126779 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1329062973911040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9342871326208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9231941683090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69325480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302336051566080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1329062973911040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9342871326208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9231941683090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69325480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302336051566080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69325478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150337242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9246055118336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69325478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292993180239872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347235 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9246055118336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302336160845312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚8749 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1329062990950400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9245713772032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1331463684957728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81011764 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347235 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34126524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81011764 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976Ïg.^]2“- _wandbruntime‚7480  _runtime‚7480ù°O ”„ 0loss/FRt1/assignment_loss‚1.7331968284911992 2loss/FBt2/assignment_loss‚0.024103691274423654 /loss/FRt1/detection_loss‚0.5676210308577634 1loss/SRqqt1/detection_loss‚0.9906598441753681 1loss/SRqqt2/detection_loss‚0.5801205611820669 _step‚720 0loss/FRt2/assignment_loss‚0.8141478491558856 2loss/SRqqt2/assignment_loss‚0.5341188823571161 3loss/FBt1/assignment_loss‚0.0028448484839172197 %loss/total_loss‚8.076142617321132 trainer/global_step‚8749 _runtime‚7480.048507796 2loss/SRqqt1/assignment_loss‚0.5216971211642194 .loss/FRt2/detection_loss‚1.126932399128617 0loss/FBt2/detection_loss‚0.40805263469418035 % _timestamp‚1.7843478213552225e+09 .loss/FBt1/detection_loss‚0.772646926356376 epoch‚21Höû«•‰ 2loss/SRqqt1/assignment_loss‚0.5216971211642194 .loss/FRt2/detection_loss‚1.126932399128617 0loss/FBt2/detection_loss‚0.40805263469418035 % _timestamp‚1.7843478213552225e+09 .loss/FBt1/detection_loss‚0.772646926356376 epoch‚21 0loss/FRt1/assignment_loss‚1.7331968284911992 2loss/FBt2/assignment_loss‚0.024103691274423654 /loss/FRt1/detection_loss‚0.5676210308577634 1loss/SRqqt1/detection_loss‚0.9906598441753681 1loss/SRqqt2/detection_loss‚0.5801205611820669 _step‚720 0loss/FRt2/assignment_loss‚0.8141478491558856 2loss/SRqqt2/assignment_loss‚0.5341188823571161 3loss/FBt1/assignment_loss‚0.0028448484839172197 %loss/total_loss‚8.076142617321132 trainer/global_step‚8749 _runtime‚7480.048507796ÐaA“–‚(:ˆ ²ùëÒç̲– gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o ¥Û—‚(:Ð ²ùëÒð‡’˜ disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 14086.125 cpu‚26.17867606878672 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 1.388957e+07 network.recv‚ 2.060599e+06 disk.rbd1.out‚ 18.33203125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚24748.8583984375" proc.memory.rssMB‚ 2724.9921875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda.out‚ 239.00390625 disk.sda2.out‚0" disk.dm-0.out‚24748.8583984375 disk.nvme0n1.in‚ 14086.125) proc.memory.percent‚16.63203239440918( proc.memory.availableMB‚ 496.69140625$ disk./.usageGB‚588.4449691772461 disk.rbd1.in‚0 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚24748.8583984375 disk.dm-0.in‚ 14086.125# memory_percent‚96.9684362411499) disk./.usagePercent‚39.49103561970044 disk.sda3.out‚ 239.00390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 239.00390625 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125M—¯¾“˜‚(:ˆ ÁùëÒ¾œæ“ gpu.0.gpu‚68.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡127♂(:× ÁùëÒºÿ¿– disk.sda3.out‚ 239.29296875 disk.rbd1.in‚0* disk./.usagePercent‚39.491037155745644 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 239.29296875" proc.memory.rssMB‚ 2724.9921875& disk.nvme0n1.out‚24753.20751953125$ disk./.usageGB‚588.4449920654297 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 14086.125 disk.rbd1.out‚ 18.39453125 cpu‚26.178265592278763 disk.dm-1.out‚ 239.29296875# disk.dm-0.out‚24753.20751953125 disk.sda2.in‚0) proc.memory.percent‚16.63203239440918 disk.sda2.out‚0 disk.nvme0n1.in‚ 14086.125$ memory_percent‚96.96977138519287 proc.cpu.threads‚17 network.sent‚ 1.3952452e+07 network.recv‚ 2.065308e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 14086.125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125( proc.memory.availableMB‚ 496.47265625( disk.nvme0c0n1.out‚24753.20751953125Ø>:š‚(:… ÐùëÒ ó·” gpu.0.gpu‚34.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¥^!Ù›‚(:Î ÐùëÒç›— disk.sda.in‚ 0.1328125 disk.sda.out‚ 239.484375 disk.sda2.out‚0 disk.sda3.out‚ 239.484375 disk.rbd1.out‚ 18.39453125 disk.sda1.in‚0 network.recv‚ 2.068688e+06 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0& disk.nvme0n1.out‚24756.92626953125 disk.sda3.in‚ 0.1328125 cpu‚26.178237303605773$ disk./.usageGB‚588.4450263977051 disk.nvme0c0n1.in‚ 14086.125( disk.nvme0c0n1.out‚24756.92626953125 disk.sda2.in‚0$ memory_percent‚96.96969985961914" proc.memory.rssMB‚ 2724.9921875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 14086.125& proc.memory.availableMB‚ 496.484375 network.sent‚ 1.3957447e+07 disk.dm-1.out‚ 239.484375 disk.nvme0n1.in‚ 14086.125 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.63203239440918 proc.cpu.threads‚17# disk.dm-0.out‚24756.92626953125) disk./.usagePercent‚39.49103945981346 disk.drbd2165.in‚0ôlàË2œ- _wandbruntime‚7520  _runtime‚7520eQwWÅ¿ trainer/global_step‚8799 $ _timestamp‚1.784347861544723e+09 _runtime‚ 7520.23820132 _step‚721 & lr-AdamW/pg1‚0.0002745380689111419 & lr-AdamW/pg2‚0.0002745380689111419v.ŽpÊžÄ & lr-AdamW/pg1‚0.0002745380689111419 & lr-AdamW/pg2‚0.0002745380689111419 trainer/global_step‚8799 $ _timestamp‚1.784347861544723e+09 _runtime‚ 7520.23820132 _step‚721Ñ€y2Ÿ- _wandbruntime‚7520  _runtime‚7520JŸD^ íO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34312865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15414801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18898064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338692313312642 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9281547116164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1329411123785728 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1329411106746368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7520.239100757 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1345573181009920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9393004820992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81467141 TGDeviceStatsŸ…Šž•Monitor.on_train_batch_start/allocation.small_pool.allocated‚81467141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784347861545475e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9295737986048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1345573181009920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1336277801402368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304024212840960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚722 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1336277784363008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69699446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338692670901892 trainer/global_step‚8799 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81457164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294631189407744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9392912049664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0WHÕYø'¡òO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34312865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15414801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18898064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338692313312642 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9281547116164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1329411123785728 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1329411106746368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7520.239100757 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1345573181009920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9393004820992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81467141 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81467141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784347861545475e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9295737986048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1345573181009920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1336277801402368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304024212840960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚722 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1336277784363008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69699446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338692670901892 trainer/global_step‚8799 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81457164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294631189407744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9392912049664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ò†äŸ2¢- _wandbruntime‚7521  _runtime‚7521dãÉ¿þ&£øM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1336425025392640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1336425025392640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81466216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9393925093376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81466216 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1345721793052160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69707078 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9282217619996 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69707080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294664636778496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9296750620160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151183287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81476207 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151173294 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304058543259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183287 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81476207 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1336425042432000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9296409273856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9296409273856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15416543 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1345721434666496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34316389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1329557617418240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9296750620160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18900119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151173294 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183287 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81476207 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1336425042432000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9296409273856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9296409273856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15416543 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1345721434666496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34316389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1329557617418240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9296750620160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18900119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151173294 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9282217619996 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69707080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294664636778496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9296750620160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151183287 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81476207 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151173294 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304058543259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088ÓïGŽ…2¥- _wandbruntime‚7521  _runtime‚7521͌˹ ¦ƒ 2loss/SRqqt1/assignment_loss‚0.4902766982904053 2loss/SRqqt2/assignment_loss‚0.5362793944145088 2loss/FBt2/assignment_loss‚0.001381089000594779 _step‚724 0loss/FRt1/assignment_loss‚1.5504758487577024 0loss/FRt2/assignment_loss‚0.8825612548933269 2loss/FBt1/assignment_loss‚0.011226547450117101 1loss/SRqqt2/detection_loss‚0.6137818801343957 /loss/FBt2/detection_loss‚0.5423971604704375 %loss/total_loss‚8.263385010456247 /loss/FRt1/detection_loss‚0.5680442156837926 /loss/FBt1/detection_loss‚0.9334046254932348 epoch‚21 trainer/global_step‚8799 _runtime‚7521.046625572 /loss/FRt2/detection_loss‚1.2100605849717965 1loss/SRqqt1/detection_loss‚0.9234957108959356 $ _timestamp‚1.784347862353321e+09 E³Z§ˆ 1loss/SRqqt2/detection_loss‚0.6137818801343957 /loss/FBt2/detection_loss‚0.5423971604704375 %loss/total_loss‚8.263385010456247 /loss/FRt1/detection_loss‚0.5680442156837926 /loss/FBt1/detection_loss‚0.9334046254932348 epoch‚21 trainer/global_step‚8799 _runtime‚7521.046625572 /loss/FRt2/detection_loss‚1.2100605849717965 1loss/SRqqt1/detection_loss‚0.9234957108959356 $ _timestamp‚1.784347862353321e+09 2loss/SRqqt1/assignment_loss‚0.4902766982904053 2loss/SRqqt2/assignment_loss‚0.5362793944145088 2loss/FBt2/assignment_loss‚0.001381089000594779 _step‚724 0loss/FRt1/assignment_loss‚1.5504758487577024 0loss/FRt2/assignment_loss‚0.8825612548933269 2loss/FBt1/assignment_loss‚0.011226547450117101ÔÖW1˜‘¨‚(:† ßùëÒ§Žž” gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.739' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5478 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êÿ¹Uà©‚(:Õ ßùëÒ·Œ– disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 18.45703125 disk.rbd0.in‚0$ disk./.usageGB‚588.4450836181641 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 239.66796875 disk.rbd0.out‚0) disk./.usagePercent‚39.49104329992647 disk.drbd2165.in‚0' proc.memory.availableMB‚ 494.1796875# disk.dm-0.out‚24761.85009765625 disk.sda.out‚ 239.66796875 disk.nvme0n1.in‚ 14086.125$ memory_percent‚96.98376655578613 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.3962453e+07 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 14086.125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2727.40234375* proc.memory.percent‚16.646742820739746 disk.dm-0.in‚ 14086.125 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 239.66796875 disk.sda1.out‚0 cpu‚26.1809674269233 network.recv‚ 2.072291e+06( disk.nvme0c0n1.out‚24761.85009765625 disk.sda2.in‚0& disk.nvme0n1.out‚24761.85009765625UÂ+2ª- _wandbruntime‚7539  _runtime‚75398æD ¦=« { 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179522395133972 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_1‚0.3562019467353821 jet/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05621923878788948 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7763354778289795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03216364607214928 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6836527585983276 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 epoch‚21 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8249432444572449 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34426572918891907 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28956401348114014 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982069730758667 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8331443071365356 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4743238687515259 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1599767804145813 *jet/accuracy_2_of_4‚0.8007382154464722 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4884028732776642 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7654512524604797 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7754809856414795 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9948416352272034 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7153220176696777 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7062581777572632 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28976011276245117 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7470852136611938 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6604539752006531 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8008025884628296 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5937701463699341 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988112449645996 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6154155135154724 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.037777069956064224 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6573494076728821 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24266360700130463 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2449827641248703 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957911372184753 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6624820828437805 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6003015637397766 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30012014508247375 +particle/sensitivity‚0.6351128816604614 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5703125 jet/accuracy_3_of_5‚1 $ _timestamp‚1.784347880957174e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.770311713218689 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03216364607214928 +jet/accuracy_3_of_4‚0.09440521895885468 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4838888943195343 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9297724366188049 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6873126029968262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6515486240386963 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29068708419799805 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7551406025886536 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7635298371315002 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.83909672498703 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29990383982658386 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5735263824462891 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9901257753372192 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22029611468315125 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.855505645275116 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6644303798675537 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9937731623649597 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688567876815796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869953989982605 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6698875427246094 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3972248435020447 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04625394195318222 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08913303911685944 +jet/accuracy_3_of_3‚0.07635024935007095 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12291329354047775 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8732951879501343 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7481755018234253 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_2‚0.7900245189666748 *jet/accuracy_1_of_3‚0.9163062572479248 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9503294825553894 +particle/specificity‚0.9108109474182129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7048035264015198 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23507468402385712 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8014763593673706 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.833824634552002 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982069730758667 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6920130848884583 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6247055530548096 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043361928313970566 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3457096517086029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.39269301295280457 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 +jet/accuracy_4_of_4‚0.00957049522548914 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5100619196891785 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.855505645275116 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8516013026237488 _step‚725 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039580874145030975 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6677119135856628 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08288650214672089 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.50041663646698 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23019683361053467 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8361678123474121 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30069124698638916 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10361555963754654 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 particle/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.51030433177948 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22114674746990204 /particle/accuracy_3_of_4‚0.9968254566192627 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957696795463562 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8904324769973755 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860661625862122 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5623385906219482 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7678649425506592 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21758300065994263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874277710914612 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04166381061077118 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32423004508018494 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3935719430446625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7661906480789185 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.699791431427002 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08011545240879059 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3581710457801819 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.596097469329834 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 'particle/f_score‚0.6476575136184692 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10227511823177338 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2295973002910614 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.039182692766189575 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7510428428649902 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833283662796 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6624820828437805 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29068708419799805 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2418578416109085 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712539553642273 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 _runtime‚7539.651433238 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2134803831577301 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7516072392463684 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948861002922058 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2752063274383545 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891104102134705 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982827305793762 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9748555421829224 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6234964728355408 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4918844699859619 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7648142576217651 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2816537022590637 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.038083117455244064 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.740468442440033 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6441327333450317 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23125407099723816 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.794965538660561 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22916103899478912 trainer/global_step‚8799*˜&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5937701463699341 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988112449645996 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6154155135154724 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.037777069956064224 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6573494076728821 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24266360700130463 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2449827641248703 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957911372184753 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6624820828437805 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6003015637397766 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30012014508247375 +particle/sensitivity‚0.6351128816604614 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5703125 jet/accuracy_3_of_5‚1 $ _timestamp‚1.784347880957174e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.770311713218689 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03216364607214928 +jet/accuracy_3_of_4‚0.09440521895885468 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4838888943195343 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9297724366188049 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6873126029968262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6515486240386963 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29068708419799805 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7551406025886536 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7635298371315002 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 5 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.83909672498703 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29990383982658386 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5735263824462891 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9901257753372192 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22029611468315125 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.855505645275116 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6644303798675537 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9937731623649597 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688567876815796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869953989982605 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6698875427246094 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3972248435020447 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04625394195318222 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08913303911685944 +jet/accuracy_3_of_3‚0.07635024935007095 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12291329354047775 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8732951879501343 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7481755018234253 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_2‚0.7900245189666748 *jet/accuracy_1_of_3‚0.9163062572479248 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9503294825553894 +particle/specificity‚0.9108109474182129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7048035264015198 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23507468402385712 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8014763593673706 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.833824634552002 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982069730758667 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6920130848884583 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6247055530548096 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043361928313970566 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3457096517086029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.39269301295280457 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 +jet/accuracy_4_of_4‚0.00957049522548914 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5100619196891785 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.855505645275116 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8516013026237488 _step‚725 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039580874145030975 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6677119135856628 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08288650214672089 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.50041663646698 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23019683361053467 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8361678123474121 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30069124698638916 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10361555963754654 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 particle/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.51030433177948 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22114674746990204 /particle/accuracy_3_of_4‚0.9968254566192627 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957696795463562 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8904324769973755 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860661625862122 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5623385906219482 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7678649425506592 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21758300065994263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874277710914612 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04166381061077118 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32423004508018494 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3935719430446625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7661906480789185 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.699791431427002 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08011545240879059 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3581710457801819 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.596097469329834 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 'particle/f_score‚0.6476575136184692 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10227511823177338 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9î`®ó_"Purity/1FRt*SRqqt*FBt/event_purity‚0.2295973002910614 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.039182692766189575 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7510428428649902 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833283662796 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6624820828437805 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29068708419799805 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2418578416109085 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712539553642273 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 _runtime‚7539.651433238 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2134803831577301 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7516072392463684 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948861002922058 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2752063274383545 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891104102134705 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982827305793762 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9748555421829224 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6234964728355408 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4918844699859619 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7648142576217651 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2816537022590637 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.038083117455244064 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.740468442440033 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6441327333450317 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23125407099723816 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.794965538660561 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22916103899478912 trainer/global_step‚8799 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179522395133972 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_1‚0.3562019467353821 jet/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05621923878788948 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7763354778289795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03216364607214928 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6836527585983276 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 epoch‚21 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8249432444572449 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34426572918891907 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28956401348114014 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982069730758667 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8331443071365356 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4743238687515259 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1599767804145813 *jet/accuracy_2_of_4‚0.8007382154464722 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4884028732776642 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7654512524604797 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7754809856414795 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9948416352272034 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1ÕÇ÷åß‘­‚(:† îùëÒÐüÝ“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚261.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( ùÇ뮂(:à îùëÒÁÌ–( disk.nvme0c0n1.out‚25010.21337890625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 239.890625$ disk.nvme0c0n1.in‚14193.95703125& disk.nvme0n1.out‚25010.21337890625 disk.rbd1.in‚0( proc.memory.availableMB‚ 741.99609375 proc.cpu.threads‚17$ disk./.usageGB‚588.4460029602051 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚14193.95703125$ memory_percent‚95.47121524810791 cpu‚26.18354341630399 network.sent‚ 1.4025318e+07 network.recv‚ 2.076564e+06 disk.sda.out‚ 239.890625) disk./.usagePercent‚39.49110499774221 disk.rbd0.out‚0 disk.sda3.out‚ 239.890625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2730.37890625# disk.dm-0.out‚25010.21337890625 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚14193.95703125 disk.rbd1.out‚ 18.45703125* proc.memory.percent‚16.664910316467285 disk.sda3.in‚ 0.1328125 þœ “¯‚(:ˆ ýùëÒµ¸Ï“ gpu.0.gpu‚58.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚160.336' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÙwÓÙ°‚(:Î ýùëÒµ‡ä• disk.nvme0n1.out‚ 25016.0625' proc.memory.availableMB‚ 741.9296875) disk./.usagePercent‚39.49111447002098$ disk./.usageGB‚588.4461441040039 disk.sda.out‚ 240.1015625 disk.sda3.out‚ 240.1015625 disk.drbd2165.in‚0 cpu‚26.183351126001636! disk.nvme0c0n1.out‚ 25016.0625 network.recv‚ 2.080828e+06 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚14193.95703125 disk.rbd1.out‚ 18.51953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚14193.95703125 disk.rbd0.in‚0 network.sent‚ 1.4082543e+07 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125$ memory_percent‚95.47162055969238# proc.memory.rssMB‚ 2730.37890625* proc.memory.percent‚16.664910316467285 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚14193.95703125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 240.1015625 disk.dm-0.out‚ 25016.0625:Bi‚(:ˆ ŒúëÒ©¶¡” gpu.0.gpu‚43.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Æ沂(:Û ŒúëÒîŽò• disk.sda1.in‚0' proc.memory.availableMB‚ 742.1328125 disk.rbd1.in‚0 network.sent‚ 1.4087667e+07( disk.nvme0c0n1.out‚25272.66162109375" disk.nvme0n1.in‚14193.95703125 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 242.03125# proc.memory.rssMB‚ 2730.37890625 disk.rbd0.out‚0$ disk./.usageGB‚588.4496879577637 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.664910316467285 disk.rbd1.out‚ 18.51953125 network.recv‚ 2.08419e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 242.03125# disk.dm-0.out‚25272.66162109375 disk.sda.out‚ 242.03125 cpu‚26.18315304433908 disk.dm-0.in‚14193.95703125& disk.nvme0n1.out‚25272.66162109375) disk./.usagePercent‚39.49135230102029$ disk.nvme0c0n1.in‚14193.95703125$ memory_percent‚95.47038078308105‹‰z…2³- _wandbruntime‚7582  _runtime‚7582„L€Ã´½ % lr-AdamW/pg1‚0.000272782152454973 % lr-AdamW/pg2‚0.000272782152454973 trainer/global_step‚8849 # _timestamp‚1.78434792335878e+09 _runtime‚7582.055455566 _step‚726úÛȵ % lr-AdamW/pg1‚0.000272782152454973 % lr-AdamW/pg2‚0.000272782152454973 trainer/global_step‚8849 # _timestamp‚1.78434792335878e+09 _runtime‚7582.055455566 _step‚726ÖÑZ2¶- _wandbruntime‚7582  _runtime‚7582ƒöó'·íO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296318539433984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305780877394944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81981943 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34537246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296318558046208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70201712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚727 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8849 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354933227851776 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1345569402494976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70201714 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1345569402494976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305780972001280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9462413955072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152183655 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19032678 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚7582.056428673 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1345569419534336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9349542778416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345569419534336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70201712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81991920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169}Ìn¹ø'¸òO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9349883328306 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15504568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34537508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1338645303554048 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1354933586230272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1347994846332464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81991920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1345569402494976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305780972001280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9462413955072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152183655 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19032678 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚7582.056428673 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1345569419534336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9349542778416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345569419534336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70201712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81991920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296318539433984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305780877394944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81981943 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34537246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296318558046208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70201712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚727 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8849 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354933227851776 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1345569402494976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70201714 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305815430142976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345716660563968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34541081 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1345716660563968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70209346 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1348142368058058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29 ª926351986804736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70209344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15506316 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9365179330048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843479239800117e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345716643524608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305815327155200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350894382140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19034768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9463443338240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9463358962688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1355081839894016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9350553832138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34541344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15506313 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9364837983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70209346 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355081839894016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81990995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296351968192512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82000986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚8849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19035028 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355081481508352 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9365179330048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0½Y#Ò'»üM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350894382140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19034768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9463443338240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9463358962688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1355081839894016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9350553832138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34541344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15506313 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9364837983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70209346 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355081839894016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81990995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296351968192512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82000986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚8849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19035028 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355081481508352 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9365179330048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152200339 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚ 7582.67378009 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305815430142976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345716660563968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34541081 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1345716660563968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70209346 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1348142368058058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296351986804736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70209344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15506316 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9365179330048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843479239800117e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345716643524608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305815327155200Øn%÷>2¼- _wandbruntime‚7582  _runtime‚7582÷Ýà ½‡ /loss/FRt1/detection_loss‚0.6055036101771919 3loss/FBt1/assignment_loss‚0.0003417583035644476 3loss/FBt2/assignment_loss‚0.0027746604866908165 1loss/SRqqt2/detection_loss‚0.6316593938112116 epoch‚22 trainer/global_step‚8849 _step‚729 /loss/FRt2/detection_loss‚1.1729858026104183 /loss/FBt2/detection_loss‚0.4480115341022611 %loss/total_loss‚7.697076840216303 _runtime‚7582.677013554 0loss/FRt1/assignment_loss‚1.5801321319662607 1loss/SRqqt1/detection_loss‚0.9096101751278522 /loss/FBt1/detection_loss‚0.5549894305877388 % _timestamp‚1.7843479239832182e+09 0loss/FRt2/assignment_loss‚0.8511303179505942 2loss/SRqqt1/assignment_loss‚0.5112388290777566 3loss/SRqqt2/assignment_loss‚0.42869919601476214~ƒž¾Œ 0loss/FRt2/assignment_loss‚0.8511303179505942 2loss/SRqqt1/assignment_loss‚0.5112388290777566 3loss/SRqqt2/assignment_loss‚0.42869919601476214 /loss/FRt1/detection_loss‚0.6055036101771919 3loss/FBt1/assignment_loss‚0.0003417583035644476 3loss/FBt2/assignment_loss‚0.0027746604866908165 1loss/SRqqt2/detection_loss‚0.6316593938112116 epoch‚22 trainer/global_step‚8849 _step‚729 /loss/FRt2/detection_loss‚1.1729858026104183 /loss/FBt2/detection_loss‚0.4480115341022611 %loss/total_loss‚7.697076840216303 _runtime‚7582.677013554 0loss/FRt1/assignment_loss‚1.5801321319662607 1loss/SRqqt1/detection_loss‚0.9096101751278522 /loss/FBt1/detection_loss‚0.5549894305877388 % _timestamp‚1.7843479239832182e+09Ùx*\“¿‚(:ˆ ›úëÒ¬¿³” gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.924' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ÝÒZíÀ‚(:â ›úëÒˆ«–$ memory_percent‚95.47216892242432 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.58203125* proc.memory.percent‚16.664910316467285 network.recv‚ 2.08761e+06) disk./.usagePercent‚39.49134154870386 disk.sda1.out‚0# proc.memory.rssMB‚ 2730.37890625 cpu‚26.18269079479593 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚14193.95703125 disk.dm-0.in‚14193.95703125 disk.sda2.out‚0" disk.nvme0n1.in‚14193.95703125& disk.nvme0n1.out‚25278.92822265625 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda.out‚ 242.7109375 disk.rbd0.out‚0 disk.dm-1.out‚ 242.7109375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25278.92822265625# disk.dm-0.out‚25278.92822265625( proc.memory.availableMB‚ 741.83984375$ disk./.usageGB‚588.4495277404785 disk.sda3.out‚ 242.7109375 network.sent‚ 1.4092714e+07 disk.dm-1.in‚ 0.1328125…’ŸžÁ‚(:“ ªúëÒº»–“ gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚182.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.574200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Õ‚(:Ê ªúëÒ›æ– network.sent‚ 1.4155691e+07 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚25281.5986328125" disk.dm-0.out‚25281.5986328125 disk.sda.out‚ 243.09375 network.recv‚ 2.091977e+06$ disk./.usageGB‚588.4495582580566 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 14283.9375# memory_percent‚95.4721450805664* proc.memory.percent‚16.664910316467285% disk.nvme0n1.out‚25281.5986328125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 243.09375 disk.nvme0c0n1.in‚ 14283.9375 disk.dm-0.in‚ 14283.9375 disk.sda3.in‚ 0.1328125% proc.memory.availableMB‚ 741.84375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2730.37890625 cpu‚26.1823285532086* disk./.usagePercent‚39.491343596764125 disk.drbd2165.out‚0 disk.sda3.out‚ 243.09375 disk.rbd1.out‚ 18.58203125Jó’Â(:‡ ¹úëÒ¨ôÏ“ gpu.0.gpu‚18.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mB˜âÄ‚(:× ¹úëÒ¥ºï• disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 14283.9375# proc.memory.rssMB‚ 2730.37890625 network.recv‚ 2.095361e+06* disk./.usagePercent‚39.491343596764125 disk.sda1.out‚0 disk.dm-1.out‚ 243.50390625& disk.nvme0n1.out‚25286.60693359375* proc.memory.percent‚16.664910316467285 cpu‚26.18217369843535 proc.cpu.threads‚17 network.sent‚ 1.4160815e+07 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25286.60693359375 disk.sda.out‚ 243.50390625 disk.sda2.in‚0 disk.sda3.out‚ 243.50390625 disk.rbd1.out‚ 18.6484375 disk.dm-0.in‚ 14283.9375 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 14283.9375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚588.4495582580566# disk.dm-0.out‚25286.60693359375 disk.rbd0.out‚0 disk.drbd2165.in‚0$ memory_percent‚95.47367095947266% proc.memory.availableMB‚ 741.59375Ú†2Å- _wandbruntime‚7623  _runtime‚7623 ÷·OÉÆà _step‚730 ' lr-AdamW/pg1‚0.00027102313739731614 ' lr-AdamW/pg2‚0.00027102313739731614 trainer/global_step‚8899 % _timestamp‚1.7843479643508043e+09 _runtime‚7623.050726019hˆ$ÎÇÈ _step‚730 ' lr-AdamW/pg1‚0.00027102313739731614 ' lr-AdamW/pg2‚0.00027102313739731614 trainer/global_step‚8899 % _timestamp‚1.7843479643508043e+09 _runtime‚7623.050726019ÚXr^2È- _wandbruntime‚7623  _runtime‚7623—û©"ÉîO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1352931471015936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1362346333213696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1345970837147648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9513843530240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8899 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82436395 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9400499815214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1345970854187008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82436395 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297989977360384 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1362346333213696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307503820890624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15591818 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1352931453976576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1355370996412972 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1362345974835200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15591821 % _timestamp‚1.7843479643580384e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1362345974835200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414862197760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400159265324 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9414520858624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153029686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82446372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82446372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34729816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583314 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.p±¦¡Kallocated‚297989995972608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9513925815808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1352931471015936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7623.051827619 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414862197760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34729546 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9414520858624þXé•ù'ÊóO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9513843530240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8899 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82436395 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9400499815214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1345970854187008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82436395 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297989977360384 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1362346333213696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307503820890624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15591818 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1352931453976576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1355370996412972 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1362345974835200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15591821 % _timestamp‚1.7843479643580384e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1362345974835200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414862197760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400159265324 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9414520858624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153029686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82446372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82446372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34729816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583314 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297989995972608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9513925815808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1352931471015936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7623.051827619 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414862197760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34729546 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9414520858624 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1352931471015936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1362346333213696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1345970837147648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15593566 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298023424731136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1355518875727928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82445447 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353078712045568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1362494586877440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34733385 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843479649733653e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15593563 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1346117347819520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70590944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1353078712045568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1362494228491776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70590944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298023406118912 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9401510869048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353078695006208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9514873060352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153046384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153036391 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82455438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34733645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9514953241600 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1362494586877440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ϪÕë'ÍüM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153036391 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82455438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34733645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9514953241600 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1362494586877440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1346117364858880 _runtime‚7623.667103138 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82455438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70590946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140079 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15593566 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298023424731136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1355518875727928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82445447 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353078712045568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1362494586877440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34733385 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843479649733653e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15593563 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1346117347819520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70590944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1353078712045568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1362494228491776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70590944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298023406118912 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9401510869048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353078695006208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9514873060352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153046384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144Ü°œl”2Î- _wandbruntime‚7623  _runtime‚7623,sP χ /loss/FBt1/detection_loss‚0.5180998902900943 %loss/total_loss‚7.622168229692698 epoch‚22 trainer/global_step‚8899 _runtime‚7623.668253329 0loss/FRt1/assignment_loss‚1.5346476091209211 1loss/SRqqt2/detection_loss‚0.5972946294655218 3loss/SRqqt2/assignment_loss‚0.47845102246110127 /loss/FRt1/detection_loss‚0.5657068641323617 0loss/FBt2/detection_loss‚0.31357669074181466 0loss/FRt2/assignment_loss‚0.9151132062267519 2loss/SRqqt1/assignment_loss‚0.5590651160851925 4loss/FBt1/assignment_loss‚0.00028831304733094906 2loss/FBt2/assignment_loss‚0.003012824037920666 /loss/FRt2/detection_loss‚1.1787143218927323 $ _timestamp‚1.784347964974954e+09 _step‚733 1loss/SRqqt1/detection_loss‚0.9581977421909544=hååÐŒ trainer/global_step‚8899 _runtime‚7623.668253329 0loss/FRt1/assignment_loss‚1.5346476091209211 1loss/SRqqt2/detection_loss‚0.5972946294655218 3loss/SRqqt2/assignment_loss‚0.47845102246110127 /loss/FRt1/detection_loss‚0.5657068641323617 0lossSÒTàþ/FBt2/detection_loss‚0.31357669074181466 0loss/FRt2/assignment_loss‚0.9151132062267519 2loss/SRqqt1/assignment_loss‚0.5590651160851925 4loss/FBt1/assignment_loss‚0.00028831304733094906 2loss/FBt2/assignment_loss‚0.003012824037920666 /loss/FRt2/detection_loss‚1.1787143218927323 $ _timestamp‚1.784347964974954e+09 _step‚733 1loss/SRqqt1/detection_loss‚0.9581977421909544 /loss/FBt1/detection_loss‚0.5180998902900943 %loss/total_loss‚7.622168229692698 epoch‚22݉)Ûc“Ñ‚(:ˆ ÈúëÒŽ¯ˆ– gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚235.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”£ÉiÞÒ‚(:Ó ÈúëÒõ¦·—) disk./.usagePercent‚39.49134590083194# disk./.usageGB‚588.449592590332 disk.rbd0.out‚0$ memory_percent‚95.47369480133057 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2730.37890625& disk.nvme0c0n1.out‚25289.353515625 disk.sda.out‚ 243.77734375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 14283.9375 disk.dm-0.in‚ 14283.9375! disk.dm-0.out‚25289.353515625 disk.sda2.in‚0$ disk.nvme0n1.out‚25289.353515625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 741.58984375* proc.memory.percent‚16.664910316467285 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 243.77734375 cpu‚26.181871242860023 network.sent‚ 1.4165669e+07 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 14283.9375 disk.drbd2165.out‚0 network.recv‚ 2.098844e+06 disk.sda3.out‚ 243.77734375 disk.rbd1.out‚ 18.6484375 disk.sda1.in‚0 disk.sda1.out‚0'\ŸËœÓ‚(:‘ ×úëÒÙ‰©” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.735599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô+aéÔ‚(:Þ ×úëÒ³–Ó—( proc.memory.availableMB‚ 741.82421875* disk./.usagePercent‚39.491288299136734 disk.dm-1.out‚ 243.9921875 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 14301.953125 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 cpu‚26.181579699077794 proc.cpu.threads‚17$ disk./.usageGB‚588.4487342834473# disk.dm-0.out‚25293.77001953125 disk.sda.in‚ 0.1328125$ memory_percent‚95.47226428985596 disk.nvme0n1.in‚ 14301.953125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.4228652e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25293.77001953125" disk.nvme0c0n1.in‚ 14301.953125 disk.sda2.out‚0& disk.nvme0n1.out‚25293.77001953125 disk.rbd1.out‚ 18.7265625 disk.sda.out‚ 243.9921875# proc.memory.rssMB‚ 2730.37890625* proc.memory.percent‚16.664910316467285 network.recv‚ 2.103349e+06 disk.sda3.out‚ 243.9921875 disk.sda1.in‚0›8‘2Õ- _wandbruntime‚7663  _runtime‚7663æ|ÈÉÖà ' lr-AdamW/pg1‚0.00026926113671642136 ' lr-AdamW/pg2‚0.00026926113671642136 trainer/global_step‚8949 % _timestamp‚1.7843480051237783e+09 _runtime‚7663.817345091 _step‚734ƒÑÎ×È _runtime‚7663.817345091 _step‚734 ' lr-AdamW/pg1‚0.00026926113671642136 ' lr-AdamW/pg2‚0.00026926113671642136 trainer/global_step‚8949 % _timestamp‚1.7843480051237783e+09Þšotï2Ø- _wandbruntime‚7663  _runtime‚7663z«’£ô'ÙîO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70964914 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82900824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153855759 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70964912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299661433899008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9450775752232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360293505458176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1360293522497536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9565236794368 _step‚735 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1369758721818624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34921372 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1362747504082730 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9451116302122 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309226652081152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1369758721818624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34921630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1353296387780608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843480051248617e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70964914 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70964912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1353296370741248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9465216360448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15679068 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1369759080197120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9565323274240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576N½‡#ù'ÚóO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15679068 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1369759080197120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9565323274240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70964914 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82900824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153855759 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70964912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299661433899008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9450775752232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360293505458176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1360293522497536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9565236794368 _step‚735 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1369758721818624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34921372 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1362747504082730 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9451116302122 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309226652081152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1369758721818624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34921630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1353296387780608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843480051248617e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70964914 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70964912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1353296370741248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9465216360448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088ßÞ+ÍÚ2Û- _wandbruntime‚7664  _runtime‚7664f%^^— ÜöM _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1353442898452480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1360440746487808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82899899 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34925217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15680813 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466570333696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299694844045312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1369907333860864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872443 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1369907333860864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153872443 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34925476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1353442881413120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153882436 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9466228987392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1362894668219074 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9452127355956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15680816 _step‚736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70972546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309261110257152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70972546 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1360440763527168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19244404 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1369906975475200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9466228987392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82909890 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299694862657536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784348005737307e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82909890 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19244660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_ð¶AËebatch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7664.43106504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1362895025808436 trainer/global_step‚8949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70972544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1360440763527168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70972544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309261215342080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9451786805954 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÅõÏœ'ÝûM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9451786805954 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1353442898452480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1360440746487808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82899899 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34925217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15680813 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466570333696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299694844045312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1369907333860864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872443 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1369907333860864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153872443 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34925476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1353442881413120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153882436 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9466228987392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1362894668219074 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9452127355956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15680816 _step‚736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70972546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309261110257152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70972546 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1360440763527168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19244404 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1369906975475200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9466228987392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82909890 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299694862657536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784348005737307e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82909890 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19244660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7664.43106504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1362895025808436 trainer/global_step‚8949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70972544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1360440763527168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70972544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309261215342080àB²f2Þ- _wandbruntime‚7664  _runtime‚7664?v½ ߇ /loss/FBt2/detection_loss‚0.3379132100674401 _step‚737 0loss/FRt2/assignment_loss‚0.9259213293640359 /loss/FRt1/detection_loss‚0.5117085972794356 1loss/SRqqt2/detection_loss‚0.6235711717840493 trainer/global_step‚8949 3loss/FBt2/assignment_loss‚0.0002781444868249714 epoch‚22 $ _timestamp‚1.784348005741715e+09 3loss/SRqqt1/assignment_loss‚0.42828638791443524 2loss/SRqqt2/assignment_loss‚0.4451814536935047 /loss/FRt2/detection_loss‚1.1975404957886921 %loss/total_loss‚7.723562726816624 _runtime‚7664.435001634 0loss/FRt1/assignment_loss‚1.6696793422649043 4loss/FBt1/assignment_loss‚2.1728403046087203e-05 1loss/SRqqt1/detection_loss‚0.8451441325464705 /loss/FBt1/detection_loss‚0.7383167332237852†®àŒ 0loss/FRt1/assignment_loss‚1.6696793422649043 4loss/FBt1/assignment_loss‚2.1728403046087203e-05 1loss/SRqqt1/detection_loss‚0.8451441325464705 /loss/FBt1/detection_loss‚0.7383167332237852 /loss/FBt2/detection_loss‚0.3379132100674401 _step‚737 0loss/FRt2/assignment_loss‚0.9259213293640359 /loss/FRt1/detection_loss‚0.5117085972794356 1loss/SRqqt2/detection_loss‚0.6235711717840493 trainer/global_step‚8949 3loss/FBt2/assignment_loss‚0.0002781444868249714 epoch‚22 $ _timestamp‚1.784348005741715e+09 3loss/SRqqt1/assignment_loss‚0.42828638791443524 2loss/SRqqt2/assignment_loss‚0.4451814536935047 /loss/FRt2/detection_loss‚1.1975404957886921 %loss/total_loss‚7.723562726816624 _runtime‚7664.435001634á„lP[›á‚(: æúëÒñ†‚” gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.496' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.89919999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D‘h³ëâ‚(:à æúëÒ³çå–$ disk./.usageGB‚588.4457359313965 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.dm-0.in‚ 14403.671875 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2730.37890625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.out‚ 245.23046875" disk.nvme0c0n1.in‚ 14403.671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 14403.671875( proc.memory.availableMB‚ 741.77734375 cpu‚26.181329153992777 network.recv‚ 2.106646e+06# disk.dm-0.out‚25301.11474609375& disk.nvme0n1.out‚25301.11474609375 disk.sda.out‚ 245.23046875* proc.memory.percent‚16.664910316467285 network.sent‚ 1.4233654e+07 disk.sda3.out‚ 245.23046875$ memory_percent‚95.47255039215088) disk./.usagePercent‚39.49108707721482 disk.rbd0.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚25301.11474609375 disk.rbd1.out‚ 18.7265625Óþ+ãžã‚(:“ õúëÒÁ›í“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.388' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.877599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âIìôáä‚(:Ö õúëÒ¡§…–# proc.memory.rssMB‚ 2730.37890625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 14403.671875 disk.sda.out‚ 246.58984375$ disk./.usageGB‚588.4454383850098 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 741.8359375 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0$ memory_percent‚95.47219276428223 proc.cpu.threads‚17) disk./.usagePercent‚39.49106710862715 disk.rbd1.in‚0 disk.dm-0.in‚ 14403.671875 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda3.out‚ 246.58984375 disk.drbd2165.out‚0 network.recv‚ 2.111487e+06 disk.rbd1.out‚ 18.7890625* proc.memory.percent‚16.664910316467285 cpu‚26.181014007795678 network.sent‚ 1.4296612e+07 disk.dm-0.out‚25307.82421875 disk.nvme0n1.in‚ 14403.671875# disk.nvme0n1.out‚25307.82421875 disk.dm-1.out‚ 246.58984375 disk.sda1.in‚0% disk.nvme0c0n1.out‚25307.82421875 disk.sda2.out‚0R㣞å‚(:“ „ûëÒê¦þ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚196.445' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.288999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÚ×ëêæ‚(:ß „ûëÒþæ‘–$ disk./.usageGB‚588.4440612792969 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 14403.671875$ memory_percent‚96.97384834289551* proc.memory.percent‚16.664910316467285 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.180791283411562 disk.sda3.out‚ 247.24609375 disk.sda.out‚ 247.24609375) disk./.usagePercent‚39.49097468990729 disk.dm-0.in‚ 14403.671875# disk.dm-0.out‚25310.45166015625 disk.nvme0n1.in‚ 14403.671875 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2730.37890625 disk.rbd0.out‚0 network.recv‚ 2.114487e+06& disk.nvme0n1.out‚25310.45166015625 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 247.24609375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25310.45166015625' proc.memory.availableMB‚ 495.8046875 network.sent‚ 1.4301637e+07 disk.sda2.in‚0 disk.rbd1.out‚ 18.7890625 disk.rbd0.in‚0 disk.sda1.in‚0>>åï2ç- _wandbruntime‚7704  _runtime‚7704…§’ÇèÁ _runtime‚7704.979435737 _step‚738 & lr-AdamW/pg1‚0.0002674962635822998 & lr-AdamW/pg2‚0.0002674962635822998 trainer/global_step‚8999 % _timestamp‚1.7843480462826993e+09…ÎçéÌéÆ _step‚738 & lr-AdamW/pg1‚0.0002674962635822998 & lr-AdamW/pg2‚0.0002674962635822998 trainer/global_step‚8999 % _timestamp‚1.7843480462826993e+09 _runtime‚7704.979435737â¼Ê©‘2ê- _wandbruntime‚7704  _runtime‚7704s Eó'ëíO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83345299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35112849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9616632683520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310949485896704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35112568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19346528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1377171827180544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301332871825408 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1367655556939776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71346514 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1377171468802048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154701790 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310949580503040 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1377171827180544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚739 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚7704.980395199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1370123654163238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19346250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1360621921374208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71346512 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1367655573979136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1367655556939776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9501732789030 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9616708677632 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1370123296573988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8999ÑPCÉ[ìòO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1367655573979136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1367655556939776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9501732789030 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9616708677632 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1370123296573988 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8999 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83345299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35112849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9616632683520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310949485896704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35112568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_bºHõ¨atch_start/inactive_split.small_pool.allocated‚19346528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1377171827180544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301332871825408 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1367655556939776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71346514 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1377171468802048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154701790 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310949580503040 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1377171827180544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚739 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚7704.980395199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1370123654163238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19346250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1360621921374208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71346512 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ã—£™2í- _wandbruntime‚7705  _runtime‚7705å”Öý&î÷M @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843480469017818e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35116395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9617745059328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516924489216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83364342 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71354146 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1370270818299582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9502403292862 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83354351 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1377320080844288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1360768432046080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71354144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516924489216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310984045643264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9517265835520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15768066 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1360768415006720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301366300583936 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35116651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19348585 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708495 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1367802815008768 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1367802815008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301366281971712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19348332 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1367802797969408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15768063 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9502743842864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71354146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83364342 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35116651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19348585 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708495 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1367802815008768 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1367802815008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301366281971712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19348332 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1367802797969408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15768063 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9502743842864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71354146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83364342 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35116395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9617745059328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516924489216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83364342 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71354146 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1370270818299582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9502403292862 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83354351 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1377320080844288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1360768432046080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71354144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516924489216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310984045643264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9517265835520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15768066 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1360768415006720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301366300583936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83809728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9668201436672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83799751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1367947454967808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1384584574163968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35304356 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15853568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19450788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537842 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35304621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83809728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15853571 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784348087103269e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9566607364096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312672511188480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1384584215785472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19451050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303004309751808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1384584215785472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83809728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9668201436672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83799751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1367947454967808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1384584574163968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35304356 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15853568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19450788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537842 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35304621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83809728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15853571 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784348087103269e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9566607364096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312672511188480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1384584215785472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19451050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303004309751808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1384584215785472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35308130 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83818794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312706967147520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303037738510336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71735744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9567619991040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83808803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19452817 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843480877153504e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83818794 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554540 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1384732469442048 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1375164849451008 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1368093965639680 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155554540 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155544547 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9669228637184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71735746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9669148455936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9049ŽUñûÈ üM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71735744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9567619991040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83808803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19452817 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843480877153504e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83818794 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554540 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1384732469442048 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1375164849451008 _JDeviceStatsMåãë;:onitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1368093965639680 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155554540 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155544547 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9669228637184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71735746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9669148455936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83808803 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71735746 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1377646968380090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35308399 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35308130 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83818794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312706967147520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303037738510336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0èU«Á2‚- _wandbruntime‚7746  _runtime‚7746=iMÒ ƒ… 1loss/SRqqt1/detection_loss‚0.8718834867516455 /loss/FBt1/detection_loss‚0.8815927751664374 /loss/FBt2/detection_loss‚0.4588392545667696 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.7317703338426964 3loss/FBt1/assignment_loss‚0.0014879731448913644 .loss/FRt2/detection_loss‚1.159957667377562 epoch‚22 2loss/SRqqt1/assignment_loss‚0.5560058029954548 2loss/FBt2/assignment_loss‚0.045739607602541706 1loss/SRqqt2/detection_loss‚0.6175185329482729 _runtime‚7746.411946819 _step‚745 %loss/total_loss‚7.891985123683714 % _timestamp‚1.7843480877186668e+09 0loss/FRt1/assignment_loss‚1.5113731947092883 3loss/SRqqt2/assignment_loss‚0.49134012659755666 /loss/FRt1/detection_loss‚0.5644763679805977å+è„Š /loss/FBt2/detection_loss‚0.4588392545667696 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.7317703338426964 3loss/FBt1/assignment_loss‚0.0014879731448913644 .loss/FRt2/detection_loss‚1.159957667377562 epoch‚22 2loss/SRqqt1/assignment_loss‚0.5560058029954548 2loss/FBt2/assignment_loss‚0.045739607602541706 1loss/SRqqt2/detection_loss‚0.6175185329482729 _runtime‚7746.411946819 _step‚745 %loss/total_loss‚7.891985123683714 % _timestamp‚1.7843480877186668e+09 0loss/FRt1/assignment_loss‚1.5113731947092883 3loss/SRqqt2/assignment_loss‚0.49134012659755666 /loss/FRt1/detection_loss‚0.5644763679805977 1loss/SRqqt1/detection_loss‚0.8718834867516455 /loss/FBt1/detection_loss‚0.8815927751664374éDŽNx“…‚(:ˆ ÀûëÒ®‡á“ gpu.0.gpu‚40.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lüy•á†‚(:Ö ÀûëÒ«„«— disk.sda2.in‚0 disk.nvme0n1.in‚ 14528.90625$ memory_percent‚96.97299003601074 disk.rbd1.out‚ 18.9140625 disk.sda1.in‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 495.9453125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2730.37890625* proc.memory.percent‚16.664910316467285 proc.cpu.threads‚17 network.sent‚ 1.4379048e+07) disk./.usagePercent‚39.49118359205523 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda.out‚ 249.234375 disk.dm-1.in‚ 0.1328125! disk.nvme0c0n1.in‚ 14528.90625 disk.sda2.out‚0 network.recv‚ 2.128638e+06 disk.dm-0.in‚ 14528.90625& disk.nvme0n1.out‚25335.29736328125 disk.sda3.out‚ 249.234375 disk.drbd2165.in‚0# disk.dm-0.out‚25335.29736328125 cpu‚26.179581966955645$ disk./.usageGB‚588.4471740722656 disk.dm-1.out‚ 249.234375( disk.nvme0c0n1.out‚25335.29736328125Ã瓇‚(:ˆ ÏûëÒÈÖö“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.326' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ä 䈂(:Ù ÏûëÒøâ„–) disk./.usagePercent‚39.49117386376893 disk.drbd2165.out‚0% disk.nvme0n1.out‚25341.7822265625 network.recv‚ 2.13335e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 249.56640625$ memory_percent‚96.97573184967041* proc.memory.percent‚16.664910316467285$ disk./.usageGB‚588.4470291137695 disk.rbd0.out‚0" disk.dm-0.out‚25341.7822265625 disk.sda3.out‚ 249.56640625( proc.memory.availableMB‚ 495.49609375 network.sent‚ 1.4441997e+07 disk.rbd1.out‚ 19.0390625 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 249.56640625 disk.sda2.out‚0 disk.nvme0n1.in‚ 14528.90625 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2730.37890625 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 14528.90625 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 14528.90625' disk.nvme0c0n1.out‚25341.7822265625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.1328125 cpu‚26.179181514296108_ÚÄY’‰‚(:‡ ÞûëÒÔÛº“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.935 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vwø슂(:á ÞûëÒºó´– disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚14626.91796875* disk./.usagePercent‚39.491172839738795 disk.drbd2165.out‚0 cpu‚26.178642944346343$ disk./.usageGB‚588.4470138549805 disk.sda3.out‚ 249.765625 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚14626.91796875( disk.nvme0c0n1.out‚25345.21630859375* proc.memory.percent‚16.664910316467285 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0& disk.nvme0n1.out‚25345.21630859375 proc.cpu.threads‚17 disk.dm-1.out‚ 249.765625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.out‚ 249.765625$ memory_percent‚96.97573184967041( proc.memory.availableMB‚ 495.49609375 network.sent‚ 1.4447063e+07 disk.rbd1.in‚0# disk.dm-0.out‚25345.21630859375# proc.memory.rssMB‚ 2730.37890625 disk.rbd1.out‚ 19.0390625 disk.sda1.out‚0 network.recv‚ 2.136669e+06 disk.sda1.in‚0 disk.dm-0.in‚14626.91796875™ªt2‹- _wandbruntime‚7786  _runtime‚7786T‡—µÇŒÁ trainer/global_step‚9099 % _timestamp‚1.7843481280307639e+09 _runtime‚7786.726473714 _step‚746 & lr-AdamW/pg1‚0.0002639583535496022 & lr-AdamW/pg2‚0.0002639583535496022q_æëÌÆ & lr-AdamW/pg2‚0.0002639583535496022 trainer/global_step‚9099 % _timestamp‚1.7843481280307639e+09 _runtime‚7786.726473714 _step‚746 & lr-AdamW/pg1‚0.0002639583535496022ê«2Ž- _wandbruntime‚7786  _runtime‚7786õw8ó'íO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9602965762846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314395401911808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1384875954324254 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1382379676942336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1382379676942336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1375272971522048 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84264180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9617644205056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84264180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109712 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1391997321147392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72109714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15940818 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304675747678208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 7786.72741976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84254203 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19555032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1384875596735004 _step‚747 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15940821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314395296819712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1391997321147392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9617302865920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 trainer/global_step‚9099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9617302865920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9617644205056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9719567753728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84254203 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375272988561408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35495853 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9719654233600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304675729065984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1391996962768896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9617302865920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9617644205056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9719567753728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84254203 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375272988561408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35495853 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9719654233600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304675729065984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1391996962768896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ëMcSš2‘- _wandbruntime‚7787  _runtime‚7787<êÿý&’÷M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382526917971968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72117344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 7787.34460179 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84263255 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9603636266678 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1382526900932608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9618315492864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314429853527040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9720586423296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1375419499233280 trainer/global_step‚9099 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84273246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156380599 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72117344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35499384 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9618656839168 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1385023118460598 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304709176436736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1382526917971968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35499653 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1392145216425472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72117346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232è“”¥'“üM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1382526900932608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9618315492864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314429853527040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9720586423296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1375419499233280 trainer/global_step‚9099 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84273246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156380599 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72117344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35499384 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9618656839168 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1385023118460598 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304709176436736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1382526917971968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35499653 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1392145216425472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72117346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382526917971968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72117344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 7787.34460179 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84263255 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9603636266678 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ìÑúøE2”- _wandbruntime‚7787  _runtime‚7787ž%µŽ •„ /loss/FBt2/detection_loss‚0.4083731681730442 epoch‚22 $ _timestamp‚1.784348128653762e+09 0loss/FRt2/assignment_loss‚0.8632650453012108 /loss/FBt1/detection_loss‚0.7638784029986709 _step‚749 0loss/FRt1/assignment_loss‚1.5538383556334354 2loss/SRqqt2/assignment_loss‚0.4501548315947771 3loss/FBt1/assignment_loss‚0.0006519575470742873 /loss/FRt2/detection_loss‚1.2528387056311263 1loss/SRqqt1/detection_loss‚0.9374576645786874 1loss/SRqqt2/detection_loss‚0.6138291489566393 trainer/global_step‚9099 _runtime‚7787.347076082 2loss/SRqqt1/assignment_loss‚0.5140546303620296 2loss/FBt2/assignment_loss‚0.011965225447985306 /loss/FRt1/detection_loss‚0.5666078865378307 %loss/total_loss‚7.9369150227625120Ñ=¤–‰ 0loss/FRt2/assignment_loss‚0.8632650453012108 /loss/FBt1/detection_loss‚0.7638784029986709 _step‚749 0loss/FRt1/assignment_loss‚1.5538383556334354 2loss/SRqqt2/assignment_loss‚0.4501548315947771 3loss/FBt1/assignment_loss‚0.0006519575470742873 /loss/FRt2/detection_loss‚1.2528387056311263 1loss/SRqqt1/detection_loss‚0.9374576645786874 1loss/SRqqt2/detection_loss‚0.6138291489566393 trainer/global_step‚9099 _runtime‚7787.347076082 2loss/SRqqt1/assignment_loss‚0.5140546303620296 2loss/FBt2/assignment_loss‚0.011965225447985306 /loss/FRt1/detection_loss‚0.5666078865378307 %loss/total_loss‚7.936915022762512 /loss/FBt2/detection_loss‚0.4083731681730442 epoch‚22 $ _timestamp‚1.784348128653762e+09íùF{+›—‚(: íûë񦐥 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚271.373' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.27459999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÄ ç˜‚(:Ü íûëҬ쇖 network.recv‚ 2.1405e+06" disk.nvme0n1.in‚14626.91796875 cpu‚26.178398934537782 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125! disk.dm-0.out‚25352.400390625 disk.sda1.out‚0* disk./.usagePercent‚39.491224297253176$ disk./.usageGB‚588.4477806091309 disk.dm-1.out‚ 250.02734375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚14626.91796875 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda3.out‚ 250.02734375 network.sent‚ 1.445206e+07 disk.dm-0.in‚14626.91796875 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚25352.400390625$ memory_percent‚96.97580337524414# proc.memory.rssMB‚ 2730.37890625 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚25352.400390625 disk.rbd1.out‚ 19.1015625 disk.rbd0.in‚0* proc.memory.percent‚16.664910316467285 disk.sda.out‚ 250.02734375& proc.memory.availableMB‚ 495.484375 proc.cpu.threads‚17&Øû™‚(:’ üûëÒü€¯“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚198.33' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.666000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03WŽEáš‚(:Ö üûëÒÖîó•$ disk./.usageGB‚588.4437942504883 disk.rbd1.in‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.sda.out‚250.375# proc.memory.rssMB‚ 2730.37890625 cpu‚26.177899509199495 disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚25357.25439453125 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚250.375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 495.56640625* proc.memory.percent‚16.664910316467285 proc.cpu.threads‚17 disk.rbd1.out‚ 19.1015625( disk.nvme0c0n1.out‚25357.25439453125" disk.nvme0n1.in‚14626.91796875$ memory_percent‚96.97530269622803 network.recv‚ 2.14467e+06 disk.sda3.out‚250.375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚14626.91796875 disk.dm-0.in‚14626.91796875# disk.dm-0.out‚25357.25439453125 disk.sda.in‚ 0.1328125 network.sent‚ 1.4517053e+07) disk./.usagePercent‚39.49095676937989Èéj2›- _wandbruntime‚7827  _runtime‚7827ª\!Ȝ ' lr-AdamW/pg1‚0.00026218554388437646 ' lr-AdamW/pg2‚0.00026218554388437646 trainer/global_step‚9149 % _timestamp‚1.7843481690857341e+09 _runtime‚ 7827.78370904 _step‚750¾hÅÇÍÇ % _timestamp‚1.7843481690857341e+09 _runtime‚ 7827.78370904 _step‚750 ' lr-AdamW/pg1‚0.00026218554388437646 ' lr-AdamW/pg2‚0.00026218554388437646 trainer/global_step‚9149îTpÁÍ2ž- _wandbruntime‚7827  _runtime‚7827ù@]ZÙŸìO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84708655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84718632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84708655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9653582249754 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84718632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1399409709752320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9667998367744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9770916006912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9668339706880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382598505115648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1399410068130816 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1389741728423936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306347185604608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1382598522155008 trainer/global_step‚9149 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1392252104404762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16028071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616Û:¢!÷' ñO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35687058 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1392251746815512 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9667998367744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1389741711384576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19658990 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72491314 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157199967 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1389741728423936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9668339706880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157209946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16028068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72491312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72491312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9668339706880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382598505115648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1399410068130816 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1389741728423936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306347185604608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1382598522155008 trainer/global_step‚9149 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1392252104404762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16028071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84708655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84718632 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84708655 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9653582249754 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84718632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1399409709752320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9667998367744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9770916006912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1382745032826880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1389888952414208 _step‚752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72498944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9669010994688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84727698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72498944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1392399626130468 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84727698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9669352340992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399558321794560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399557963408896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157216651 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316152546183680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84717707 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306380614363136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382745015787520 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35690819 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306380595750912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226644 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1392399626130468 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84727698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9669352340992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399558321794560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399557963408896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157216651 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316152546183680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84717707 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306380614363136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382745015787520 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35690819 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306380595750912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226644 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batÍj)-fch_end/requested_bytes.large_pool.allocated‚1382745032826880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1389888952414208 _step‚752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72498944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9669010994688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84727698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72498944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144ð[Jv72¤- _wandbruntime‚7828  _runtime‚7828îŸÕ ¥† %loss/total_loss‚7.947414270242915 trainer/global_step‚9149 % _timestamp‚1.7843481697067947e+09 .loss/FRt1/detection_loss‚0.575382815072165 1loss/SRqqt2/detection_loss‚0.6431214860288708 0loss/FBt1/detection_loss‚0.46243246709927915 /loss/FBt2/detection_loss‚0.4443443149856398 epoch‚22 0loss/FRt2/assignment_loss‚0.9094196138959942 2loss/SRqqt1/assignment_loss‚0.5471432027645885 4loss/FBt1/assignment_loss‚2.3178272246404674e-05 2loss/FBt2/assignment_loss‚0.017685053508964915 /loss/FRt2/detection_loss‚1.2825834968740844 1loss/SRqqt1/detection_loss‚0.8712674104523224 _runtime‚7828.400072991 _step‚753 0loss/FRt1/assignment_loss‚1.6456429912048636 2loss/SRqqt2/assignment_loss‚0.5483682400838962þ( ©¦‹ 2loss/SRqqt1/assignment_loss‚0.5471432027645885 4loss/FBt1/assignment_loss‚2.3178272246404674e-05 2loss/FBt2/assignment_loss‚0.017685053508964915 /loss/FRt2/detection_loss‚1.2825834968740844 1loss/SRqqt1/detection_loss‚0.8712674104523224 _runtime‚7828.400072991 _step‚753 0loss/FRt1/assignment_loss‚1.6456429912048636 2loss/SRqqt2/assignment_loss‚0.5483682400838962 %loss/total_loss‚7.947414270242915 trainer/global_step‚9149 % _timestamp‚1.7843481697067947e+09 .loss/FRt1/detection_loss‚0.575382815072165 1loss/SRqqt2/detection_loss‚0.6431214860288708 0loss/FBt1/detection_loss‚0.46243246709927915 /loss/FBt2/detection_loss‚0.4443443149856398 epoch‚22 0loss/FRt2/assignment_loss‚0.9094196138959942ñ1Vs³“§‚(:ˆ ‹üëÒéï ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¿’ð¨‚(:å ‹üëÒÙ³í– disk.dm-1.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 250.78515625 disk.sda3.in‚ 0.1328125 cpu‚26.177712507542164) disk./.usagePercent‚39.49121277691413 disk.sda1.out‚0 disk.dm-0.in‚14626.91796875# disk.dm-0.out‚25366.04833984375" disk.nvme0n1.in‚14626.91796875& disk.nvme0n1.out‚25366.04833984375# proc.memory.rssMB‚ 2730.37890625$ disk./.usageGB‚588.4476089477539$ disk.nvme0c0n1.in‚14626.91796875' proc.memory.availableMB‚ 495.5703125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 250.78515625 disk.dm-1.out‚ 250.78515625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 network.sent‚ 1.4519975e+07 disk.rbd0.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚25366.04833984375 network.recv‚ 2.148116e+06 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚96.97527885437012* proc.memory.percent‚16.664910316467285 disk.rbd1.out‚ 19.1640625gùÄ‘©‚(:† šüëÒêá“• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýí%Åðª‚(:å šüëÒÚöÁ– disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 250.93359375 disk.dm-1.out‚ 250.93359375$ memory_percent‚96.97549343109131# proc.memory.rssMB‚ 2730.37890625 network.recv‚ 2.152288e+06( disk./.usagePercent‚39.4912027926203 disk.sda1.in‚0( proc.memory.availableMB‚ 495.53515625* proc.memory.percent‚16.664910316467285 disk.rbd1.out‚ 19.1640625 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.drbd2165.in‚0 cpu‚26.177693882828077 proc.cpu.threads‚17 network.sent‚ 1.4583074e+07$ disk./.usageGB‚588.4474601745605 disk.rbd1.in‚0# disk.dm-0.out‚25369.88427734375" disk.nvme0n1.in‚14726.11328125 disk.sda3.in‚ 0.1328125$ disk.nvme0c0n1.in‚14726.11328125( disk.nvme0c0n1.out‚25369.88427734375 disk.dm-0.in‚14726.11328125 disk.sda.out‚ 250.93359375 disk.sda2.in‚0& disk.nvme0n1.out‚25369.88427734375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0]À«‚(:’ ©üëÒÈ“ gpu.0.gpu‚40.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äåÌÜð¬‚(:å ©üëÒôŒà–$ disk./.usageGB‚588.4475860595703 disk.sda2.in‚0* proc.memory.percent‚16.664910316467285 disk.rbd0.out‚0 network.sent‚ 1.4587944e+07 network.recv‚ 2.156201e+06 disk.sda.out‚ 251.32421875" disk.nvme0n1.in‚14726.11328125 cpu‚26.177195841336555( disk.nvme0c0n1.out‚25374.65869140625# proc.memory.rssMB‚ 2730.37890625 disk.dm-1.out‚ 251.32421875# disk.dm-0.out‚25374.65869140625 disk.rbd1.out‚ 19.2265625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚14726.11328125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 251.32421875 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.49121124086893 disk.sda1.out‚0$ memory_percent‚96.97375297546387' proc.memory.availableMB‚ 495.8203125 disk.drbd2165.in‚0 disk.dm-0.in‚14726.11328125& disk.nvme0n1.out‚25374.65869140625 disk.rbd0.in‚0éÉ\o2­- _wandbruntime‚7868  _runtime‚7868aC÷†É®Ã _runtime‚7868.487572882 _step‚754 ' lr-AdamW/pg1‚0.00026041031621802966 ' lr-AdamW/pg2‚0.00026041031621802966 trainer/global_step‚9199 % _timestamp‚1.7843482097908247e+09Þ®åmÎ¯È ' lr-AdamW/pg2‚0.00026041031621802966 trainer/global_step‚9199 % _timestamp‚1.7843482097908247e+09 _runtime‚7868.487572882 _step‚754 ' lr-AdamW/pg1‚0.00026041031621802966òᯉ2°- _wandbruntime‚7868  _runtime‚78682‰7ªô'±îO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1397103779905536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9704198736662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7843482097947276e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317840706419712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚7868.488428006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72872914 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85163107 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158045998 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1406822815114240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9718693869568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1389924038709248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚9199 _step‚755 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9703858186772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9821996408832 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1399628254485270 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1397103762866176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72872912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85163107 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1406822815114240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1397103779905536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35878468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19762885 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018604918784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1397103762866176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72872912 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85173084 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9719035208704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9718693869568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72872914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158036019 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1406822456735744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85173084 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35878203 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317840601327616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9719035208704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16115321 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308018623531008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1406822456735744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19763147 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9822082888704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16115318 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1389924055748608‰°ù'²óO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317840706419712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚7868.488428006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72872914 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85163107 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158045998 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1406822815114240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9718693869568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1389924038709248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚9199 _step‚755 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9703858186772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9821996408832 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1399628254485270 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1397103762866176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72872912 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85163107 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1406822815114240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1397103779905536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35878468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19762885 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018604918784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1397103762866176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72872912 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85173084 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9719035208704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9718693869568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72872914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158036019 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1406822456735744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85173084 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35878203 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317840601327616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9719035208704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16115321 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308018623531008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1406822456735744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19763147 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9822082888704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16115318 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1389924055748608 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1397103779905536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9704198736662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7843482097947276e+09ó²VÑ2³- _wandbruntime‚7869  _runtime‚7869Haƹ"´øM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35882078 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158052703 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72880546 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1397251020935168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1397251003895808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843482104085171e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9720047842816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72880544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823106574848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85172159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317875158864384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1390070566420480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9719706496512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.al˜Ál,Ülocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1399775776210976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚9199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85172159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9720047842816 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1406971068777984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9823022199296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1397251020935168 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9705209790496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9704869240494 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158062696 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1406971068777984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1406970710392320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1399775418621614 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1397251003895808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35882327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16117063 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269@l0 'µýM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1397251020935168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1397251003895808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843482104085171e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9720047842816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72880544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823106574848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85172159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317875158864384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1390070566420480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9719706496512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1399775776210976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚9199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85172159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9720047842816 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1406971068777984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9823022199296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1397251020935168 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9705209790496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9704869240494 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158062696 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1406971068777984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1406970710392320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1399775418621614 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1397251003895808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35882327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16117063 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35882078 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158052703 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72880546 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1141413077712059 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213742017745972 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.730322539806366 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07225826382637024 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2642960548400879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9958363771438599 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5822272300720215 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7562763690948486 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871708154678345 (particle/accuracy‚0.8476834893226624 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8661882281303406 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999250054359436 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8846763372421265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956110715866089 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2630833089351654 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5015263557434082 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 *jet/accuracy_2_of_2‚0.1555473357439041 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877514839172363 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03266787901520729 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5703981518745422 *jet/accuracy_1_of_3‚0.9148108959197998 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2526401877403259 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0967695340514183 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20032528042793274 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996357798576355 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8902872204780579 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2518864572048187 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9677729606628418 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8852230906486511 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8843348622322083 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6390408277511597 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13213999569416046 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2228582203388214 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _runtime‚ 7887.97183971 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.633616030216217 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965437054634094 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5711073875427246 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896197319030762 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7039685249328613 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48695865273475647 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7430468797683716 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33302342891693115 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2184903472661972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.94764643907547 +jet/accuracy_3_of_4‚0.09586150199174881 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9954497814178467 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3003414273262024 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6956177353858948 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 % _timestamp‚1.7843482292776356e+09 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5637989640235901 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7791473865509033 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_4_of_4‚0.9376986622810364 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6748211979866028 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.457780659198761 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030447307974100113 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02843448705971241 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21917065978050232 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6615939736366272 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8750945329666138 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411776781082153 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7038726806640625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20697957277297974 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19185101985931396 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03374322131276131 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _step‚758 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7285652756690979 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7904499769210815 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 +jet/accuracy_3_of_3‚0.07775238156318665 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49120402336120605 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7772412896156311 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8862651586532593 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6982198357582092 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970016479492188 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02843448705971241 *jet/accuracy_1_of_1‚0.3374396860599518 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8671550750732422 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8671550750732422 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3106358051300049 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905350208282471 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6455663442611694 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38221439719200134 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6559596657752991 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6460610628128052 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48314017057418823 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5541666746139526 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2359698861837387 *jet/accuracy_2_of_4‚0.8178192377090454 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.6278563141822815 epoch‚22 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3355904221534729 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2186463177204132 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚7.801915862801737 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38520246744155884 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6455663442611694 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7422181367874146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6435433626174927 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7846383452415466 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_2‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6536639928817749 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03423790633678436 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7914794683456421 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20623064041137695 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.681593120098114 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6302687525749207 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8071807622909546 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6449170708656311 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8647015690803528 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21941661834716797 —R£7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051450081169605255 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26441657543182373 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2526401877403259 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11447001993656158 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21739573776721954 ,jet/accuracy_4_of_4‚0.013899499550461769 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6validation_average_jet_accuracy‚0.3768243193626404 trainer/global_step‚9199 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6964194774627686 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.749728798866272 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46849608421325684 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6816584467887878 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033016741275787354 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20164620876312256²[-µ=½Š{ epoch‚22 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3355904221534729 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2186463177204132 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚7.801915862801737 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38520246744155884 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6455663442611694 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7422181367874146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6435433626174927 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7846383452415466 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_2‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6536639928817749 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03423790633678436 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7914794683456421 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20623064041137695 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.681593120098114 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6302687525749207 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8071807622909546 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6449170708656311 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8647015690803528 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21941661834716797 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.051450081169605255 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26441657543182373 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2526401877403259 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11447001993656158 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21739573776721954 ,jet/accuracy_4_of_4‚0.013899499550461769 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6validation_average_jet_accuracy‚0.3768243193626404 trainer/global_step‚9199 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6964194774627686 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.749728798866272 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46849608421325684 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6816584467887878 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033016741275787354 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20164620876312256 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7233709096908569 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48842594027519226 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6395931839942932 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7312194108963013 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.243694007396698 *jet/accuracy_1_of_2‚0.7746278047561646 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6747725605964661 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2627764940261841 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038982804864645004 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.037298303097486496 +particle/specificity‚0.9078076481819153 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.059212248772382736 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24422666430473328 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5758383870124817 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7488433718681335 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1141413077712059 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213742017745972 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.730322539806366 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07225826382637024 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2642960548400879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9958363771438599 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5822272300720215 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7562763690948486 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871708154678345 (particle/accuracy‚0.8476834893226624 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8661882281303406 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999250054359436 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8846763372421265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956110715866089 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2630833089351654 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5015263557434082 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 *jet/accuracy_2_of_2‚0.1555473357439041 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877514839172363 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03266787901520729 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5703981518745422 *jet/accuracy_1_of_3‚0.9148108959197998 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2526401877403259 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0967695340514183 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20032528042793274 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996357798576355 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8902872204780579 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2518864572048187 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9677729606628418 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8852230906486511 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8843348622322083 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6390408277511597 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13213999569416046 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2228582203388214 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _runtime‚ 7887.97183971 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.633616030216217 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965437054634094 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5711073875427246 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896197319030762 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7039685249328613 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48695865273475647 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7430468797683716 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33302342891693115 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2184903472661972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.94764643907547 +jet/accuracy_3_of_4‚0.09586150199174881 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9954497814178467 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3003414273262024 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6956177353858948 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 % _timestamp‚1.7843482292776356e+09 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5637989640235901 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7791473865509033 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_4_of_4‚0.9376986622810364 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6748211979866028 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.457780659198761 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030447307974100113 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02843448705971241 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21917065978050232 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6615939736366272 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8750945329666138 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7411776781082153 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7038726806640625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20697957277297974 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19185101985931396 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03374322131276131 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _step‚758 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7285652756690979 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7904499769210815 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 +jet/accuracy_3_of_3‚0.07775238156318665 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49120402336120605 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7772412896156311 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8862651586532593 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6982198357582092 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970016479492188 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02843448705971241 *jet/accuracy_1_of_1‚0.3374396860599518 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8671550750732422 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8671550750732422 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3106358051300049 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905350208282471 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6455663442611694 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38221439719200134 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6559596657752991 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6460610628128052 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48314017057418823 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5541666746139526 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2359698861837387 *jet/accuracy_2_of_4‚0.8178192377090454 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.6278563141822815önÀ‘’¾‚(:‡ ÇüëÒαä“ gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÞ@ð¿‚(:å ÇüëÒïØù– disk.sda1.out‚0 disk.dm-1.out‚ 252.44921875 disk.dm-0.in‚14726.11328125* proc.memory.percent‚16.696619987487793 cpu‚26.182742461393257 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 490.21484375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚14726.11328125 network.recv‚ 2.164343e+06 disk.sda2.in‚0" disk.nvme0n1.in‚14726.11328125$ memory_percent‚97.00796604156494 disk.sda2.out‚0 network.sent‚ 1.466887e+07) disk./.usagePercent‚39.49120816877852 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 19.2890625( disk.nvme0c0n1.out‚25383.70947265625# proc.memory.rssMB‚ 2735.57421875 proc.cpu.threads‚17$ disk./.usageGB‚588.4475402832031 disk.sda3.out‚ 252.44921875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚25383.70947265625 disk.sda.out‚ 252.44921875& disk.nvme0n1.out‚25383.70947265625 disk.drbd2165.in‚0×ÌÂ’À‚(:‡ ÖüëÒØ”­• gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚220.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.071 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ñ“ñÁ‚(:æ ÖüëÒ—™ disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.49120842478605 disk.sda2.out‚0 disk.sda3.out‚ 252.63671875 disk.sda1.out‚0 disk.dm-1.out‚ 252.63671875 network.recv‚ 2.168452e+06" disk.nvme0n1.in‚14854.19140625 disk.rbd1.in‚0# disk.dm-0.out‚25389.63330078125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 252.63671875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚14854.19140625( disk.nvme0c0n1.out‚25389.63330078125 disk.dm-0.in‚14854.19140625$ memory_percent‚97.00748920440674( proc.memory.availableMB‚ 490.29296875# proc.memory.rssMB‚ 2735.60546875* proc.memory.percent‚16.696810722351074 cpu‚26.182665339782897$ disk./.usageGB‚588.4475440979004 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.rbd1.out‚ 19.2890625 disk.rbd0.in‚0 disk.drbd2165.in‚0 network.sent‚ 1.4713238e+07& disk.nvme0n1.out‚25389.63330078125«ÛGžÂ‚(:“ åüëÒªÓŸ” gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ ŒÝìÂ(:á åüëÒØÑÜ• disk.sda1.in‚0 disk.sda2.out‚0$ disk./.usageGB‚588.4475555419922$ disk.nvme0c0n1.in‚14854.19140625( disk.nvme0c0n1.out‚25392.46533203125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚25392.46533203125# memory_percent‚97.0090389251709 cpu‚26.182249931679042 disk.sda.in‚ 0.1328125 disk.sda2.in‚0" disk.nvme0n1.in‚14854.19140625" proc.memory.rssMB‚ 2735.6328125 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚25392.46533203125' proc.memory.availableMB‚ 490.0390625 disk.drbd2165.out‚0* disk./.usagePercent‚39.491209192808654 disk.rbd1.out‚ 19.3515625 disk.dm-1.out‚ 252.8984375 disk.dm-0.in‚14854.19140625 disk.sda.out‚ 252.8984375 disk.sda3.out‚ 252.8984375 disk.rbd0.out‚0 disk.sda3.in‚ 0.1328125 network.sent‚ 1.4718419e+07 network.recv‚ 2.172195e+06 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.696977615356445 þÄ2Ä- _wandbruntime‚7928  _runtime‚79289 vŽÉÅà _runtime‚7928.223371555 _step‚759 ' lr-AdamW/pg1‚0.00025863278457011776 ' lr-AdamW/pg2‚0.00025863278457011776 trainer/global_step‚9249 % _timestamp‚1.7843482695297368e+09ÕwãÎÆÈ trainer/global_step‚9249 % _timestamp‚1.7843482695297368e+09 _runtime‚7928.223371555 _step‚759 ' lr-AdamW/pg1‚0.00025863278457011776 ' lr-AdamW/pg2‚0.00025863278457011776÷TögŽ2Ç- _wandbruntime‚7928  _runtime‚7928S4Òô'ÈîO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319597472195072 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406395398037504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1416182503575040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19897060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787122576896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319597369200128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159063059 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9787463916032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19897315 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1416182503575040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1408930787505087 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843482695306532e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73375178 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406395380998144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1416182861953536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9891498637824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406395380998144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9772194398909 trainer/global_step‚9249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9891414255104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73375180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85687881 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1416182503575040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.allåÂ' '.allocated‚1408930787505087 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843482695306532e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73375178 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406395380998144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1416182861953536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9891498637824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406395380998144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9772194398909 trainer/global_step‚9249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9891414255104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73375180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85687881 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319597472195072 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406395398037504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1416182503575040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19897060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787122576896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319597369200128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159063059 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9787463916032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19897315 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269øÖ+ð2Ê- _wandbruntime‚7928  _runtime‚79289áý&Ë÷M H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159089736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚9249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73382810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309739402315776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73382812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85706924 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1409077951641431 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73382812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36105918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1409078309230793 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319631928698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9892526382592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159079743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85706924 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚7928.842444982 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1409077951641431 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73382812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36105918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1409078309230793 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319631928698368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9892526382592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159079743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85706924 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚7928.842444982 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73382810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309739402315776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73382812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85706924 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ù-ÃAê2Í- _wandbruntime‚7928  _runtime‚7928p ìì΂ _runtime‚7928.845271308 epoch‚23 1loss/FBt1/assignment_loss‚0.03381895327335194 0loss/SRqqt2/detection_loss‚0.648826850271994 % _timestamp‚1.7843482701518743e+09 3loss/SRqqt2/assignment_loss‚0.43473256794638665 /loss/FBt1/detection_loss‚0.7954861453035846 trainer/global_step‚9249 _step‚762 .loss/FBt2/detection_loss‚0.515956573400539 %loss/total_loss‚7.905069834694981 /loss/FRt1/assignment_loss‚1.616033710904168 0loss/FRt2/assignment_loss‚0.7963530029005864 3loss/SRqqt1/assignment_loss‚0.45192080475203145 2loss/FBt2/assignment_loss‚0.001120440176968939 /loss/FRt2/detection_loss‚1.2635027562417984 1loss/SRqqt1/detection_loss‚0.7953622819304802 /loss/FRt1/detection_loss‚0.5519557475930909{:§ χ epoch‚23 1loss/FBt1/assignment_loss‚0.03381895327335194 0loss/SRqqt2/detection_loss‚0.648826850271994 % _timestamp‚1.7843482701518743e+09 3loss/SRqqt2/assignment_loss‚0.43473256794638665 /loss/FBt1/detection_loss‚0.7954861453035846 trainer/global_step‚9249 _step‚762 .loss/FBt2/detection_loss‚0.515956573400539 %loss/total_loss‚7.905069834694981 /loss/FRt1/assignment_loss‚1.616033710904168 0loss/FRt2/assignment_loss‚0.7963530029005864 3loss/SRqqt1/assignment_loss‚0.45192080475203145 2loss/FBt2/assignment_loss‚0.001120440176968939 /loss/FRt2/detection_loss‚1.2635027562417984 1loss/SRqqt1/detection_loss‚0.7953622819304802 /loss/FRt1/detection_loss‚0.5519557475930909 _runtime‚7928.845271308ú¦òü‘Ђ(:† ôüëÒæν“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚192.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.4132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È™o¬Ñ‚(:å ôüëÒÉ‹ÿ–$ memory_percent‚97.00930118560791 cpu‚26.182086388890973 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚588.4476699829102 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚14854.20703125 proc.cpu.threads‚17 network.sent‚ 1.4723289e+07 disk.rbd1.out‚ 19.3515625$ disk.nvme0c0n1.in‚14854.20703125" disk.nvme0n1.in‚14854.20703125& disk.nvme0n1.out‚25396.88818359375 disk.sda1.in‚0( proc.memory.availableMB‚ 489.99609375) disk./.usagePercent‚39.49121687303468 network.recv‚ 2.175371e+06 dñȨaDisk.sda3.out‚ 253.26171875 disk.rbd1.in‚0# disk.dm-0.out‚25396.88818359375 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.697168350219727 disk.sda3.in‚ 0.1328125 disk.sda.out‚ 253.26171875" proc.memory.rssMB‚ 2735.6640625 disk.dm-1.out‚ 253.26171875( disk.nvme0c0n1.out‚25396.88818359375°%Ç®“Ò‚(:ˆ ƒýëÒ¥†ê“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.9046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙªGgìÓ‚(:á ƒýëÒÅž–* proc.memory.percent‚16.697263717651367 disk.dm-1.out‚ 253.74609375 network.sent‚ 1.4786455e+07' disk.nvme0c0n1.out‚25400.2412109375 disk.dm-0.in‚14854.20703125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2735.6796875 cpu‚26.181750125995432 proc.cpu.threads‚17* disk./.usagePercent‚39.491225321283316" disk.nvme0n1.in‚14854.20703125% disk.nvme0n1.out‚25400.2412109375 disk.sda3.in‚ 0.1328125 disk.sda2.in‚0 disk.sda1.out‚0" disk.dm-0.out‚25400.2412109375 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.00956344604492& proc.memory.availableMB‚ 489.953125 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 253.74609375 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚14854.20703125 disk.sda.out‚ 253.74609375 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚ 2.180207e+06 disk.rbd1.out‚ 19.4140625$ disk./.usageGB‚588.4477958679199 disk.drbd2165.in‚0 disk.rbd0.in‚0côÀüžÔ‚(:“ ’ýëÒ˜ñª“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š!Ñ‚éÕ‚(:Þ ’ýëÒ§¬Õ– disk.sda.in‚ 0.1328125" proc.memory.rssMB‚ 2735.6796875 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 254.01171875 disk.rbd1.out‚ 19.4140625 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0& disk.nvme0n1.out‚25405.39111328125 disk.rbd1.in‚0 network.sent‚ 1.4791432e+07( disk./.usagePercent‚39.4912230172155$ disk./.usageGB‚588.4477615356445 disk.dm-1.out‚ 254.01171875 disk.dm-0.in‚ 14907.34375# disk.dm-0.out‚25405.39111328125* proc.memory.percent‚16.697263717651367 network.recv‚ 2.183568e+06 disk.sda.out‚ 254.01171875 disk.sda2.out‚0$ memory_percent‚97.00956344604492 cpu‚26.181155032305828 disk.sda2.in‚0" disk.nvme0n1.in‚14907.08203125& proc.memory.availableMB‚ 489.953125# disk.nvme0c0n1.in‚ 14906.6484375( disk.nvme0c0n1.out‚25405.39111328125UÏuý2Ö- _wandbruntime‚7969  _runtime‚7969 ÚQ‹É×à _step‚763 ' lr-AdamW/pg1‚0.00025685306310817675 ' lr-AdamW/pg2‚0.00025685306310817675 trainer/global_step‚9299 % _timestamp‚1.7843483104618611e+09 _runtime‚7969.157457438J¾\ùÎØÈ ' lr-AdamW/pg1‚0.00025685306310817675 ' lr-AdamW/pg2‚0.00025685306310817675 trainer/global_step‚9299 % _timestamp‚1.7843483104618611e+09 _runtime‚7969.157457438 _step‚763û°cøF2Ù- _wandbruntime‚7969  _runtime‚7969ÛHô'ÚîO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36292330 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1416306579996345 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1423595250558464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9838159417856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1423595608936960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16292341 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9837818078720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1413757432479744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73756780 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1413757449519104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311377411483648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843483104647427e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86142333 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86152310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1413757432479744 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1406483786149888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1423595250558464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9838159417856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1423595608936960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16292341 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9837818078720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1413757432479744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73756780 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1413757449519104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311377411483648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843483104647427e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86142333 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86152310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1413757432479744 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1406483786149888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159915795 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1416454101721939 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1413904673509376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1413904690548736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159925788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86151385 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1423743862600704 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1423743504215040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86161376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86161376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311410821629952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1413904690548736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73764412 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1416454459311301 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73764412 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925788 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9943684834304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1406630296821760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9839172051968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784348311081524e+09 _runtime‚7969.775276777 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321354506464256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9824162489541 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9839172051968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36296116 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36295842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73764410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16294083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9823821939539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9838830705664 H:DeviceStatsMonitor.on_train_Z?«¼batch_end/allocation.all.freed‚ 159915795 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16294086 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73764410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311410840242176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20001759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86161376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86161376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311410821629952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1413904690548736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73764412 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1416454459311301 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73764412 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925788 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9943684834304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1406630296821760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9839172051968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784348311081524e+09 _runtime‚7969.775276777 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321354506464256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9824162489541 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9839172051968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36296116 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36295842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73764410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16294083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9823821939539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9838830705664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159915795 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16294086 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73764410 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311410840242176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20001759 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159915795 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1416454101721939 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1413904673509376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ýXÊÓ2ß- _wandbruntime‚7969  _runtime‚7969 Ø…s à… /loss/FBt2/detection_loss‚0.5784165261890739 _step‚766 %loss/total_loss‚7.871374151798447 2loss/SRqqt1/assignment_loss‚0.5094677271403629 2loss/SRqqt2/assignment_loss‚0.4722674778362522 3loss/FBt1/assignment_loss‚4.334532245242606e-05 /loss/FRt1/detection_loss‚0.6009566641431773 0loss/SRqqt1/detection_loss‚0.958101882038567 1loss/SRqqt2/detection_loss‚0.5900744419307016 trainer/global_step‚9299 0loss/FRt2/assignment_loss‚0.8455148532044932 /loss/FRt2/detection_loss‚1.2258929237475589 0loss/FBt1/detection_loss‚0.46206178929423913 epoch‚23 $ _timestamp‚1.784348311085248e+09 _runtime‚7969.778622644 0loss/FRt1/assignment_loss‚1.6275366417947486 3loss/FBt2/assignment_loss‚0.0010398791568197226äìðåአ/loss/FBt2/detection_loss‚0.5784165261890739 _step‚766 %loss/total_loss‚7.871374151798447 2loss/SRqqt1/assignment_loss‚0.5094677271403629 2loss/SRqqt2/assignment_loss‚0.4722674778362522 3loss/FBt1/assignment_loss‚4.334532245242606e-05 /loss/FRt1/detection_loss‚0.6009566641431773 0loss/SRqqt1/detection_loss‚0.958101882038567 1loss/SRqqt2/detection_loss‚0.5900744419307016 trainer/global_step‚9299 0loss/FRt2/assignment_loss‚0.8455148532044932 /loss/FRt2/detection_loss‚1.2258929237475589 0loss/FBt1/detection_loss‚0.46206178929423913 epoch‚23 $ _timestamp‚1.784348311085248e+09 _runtime‚7969.778622644 0loss/FRt1/assignment_loss‚1.6275366417947486 3loss/FBt2/assignment_loss‚0.0010398791568197226þBámâ‚(:’ ¡ýëÒöç©” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Keðã‚(:å ¡ýëÒõ´£– disk.sda3.out‚ 254.37109375$ disk.nvme0c0n1.in‚14937.62890625 disk.rbd1.out‚ 19.4765625 proc.cpu.threads‚17 network.sent‚ 1.4796404e+07 disk.dm-1.out‚ 254.37109375) proc.memory.percent‚16.69785976409912 disk.sda.out‚ 254.37109375# proc.memory.rssMB‚ 2735.77734375 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 cpu‚26.180880807696305* disk./.usagePercent‚39.491187432168246$ disk./.usageGB‚588.4472312927246 disk.drbd2165.out‚0" disk.nvme0n1.in‚14937.62890625 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.dm-0.in‚14937.62890625# disk.dm-0.out‚25408.52001953125 disk.rbd1.in‚0 network.recv‚ 2.186844e+06& disk.nvme0n1.out‚25408.52001953125 disk.sda1.out‚0# memory_percent‚97.0111608505249( proc.memory.availableMB‚ 489.69140625( disk.nvme0c0n1.out‚25408.52001953125 disk.sda2.in‚0 disk.sda2.out‚0í- “ä‚(:ˆ °ýëÒ²–Æ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÏLìäå‚(:Ù °ýëÒ˜Š˜— disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚25413.25634765625 proc.memory.rssMB‚ 2735.84375* disk./.usagePercent‚39.490943200980574 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 254.59375' proc.memory.availableMB‚ 489.3359375 network.recv‚ 2.191336e+06( disk.nvme0c0n1.out‚25413.25634765625 cpu‚26.18047415669886 network.sent‚ 1.4859475e+07" disk.nvme0n1.in‚14937.62890625 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 disk.dm-0.in‚14937.62890625# disk.dm-0.out‚25413.25634765625 disk.rbd1.out‚ 19.4765625 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 254.59375 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚14937.62890625 disk.sda.out‚ 254.59375$ disk./.usageGB‚588.4435920715332 disk.sda2.in‚0$ memory_percent‚97.01333045959473* proc.memory.percent‚16.698265075683594B½z 2æ- _wandbruntime‚8009  _runtime‚8009ä~+ÏÇçÁ % _timestamp‚1.7843483512098372e+09 _runtime‚8009.906412949 _step‚767 & lr-AdamW/pg1‚0.0002550712661403905 & lr-AdamW/pg2‚0.0002550712661403905 trainer/global_step‚9349ÐŽxmÌèÆ trainer/global_step‚9349 % _timestamp‚1.7843483512098372e+09 _runtime‚8009.906412949 _step‚767 & lr-AdamW/pg1‚0.0002550712661403905 & lr-AdamW/pg2‚0.0002550712661403905ÿ§÷2é- _wandbruntime‚8009  _runtime‚8009Qô'êîO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36482398 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1413809302704128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86596785 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36482129 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843483512164965e+09 _step‚768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323042841682432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9993905792512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚9349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86606762 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1421119501000704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9888854919680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74138380 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9888513580544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735163 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74138378 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160745142 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1421119483961344 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423682730076853 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9888513580544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323042736590336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74138380 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20102807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1431007997541888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9888854919680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1421119501000704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9993992272384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1413809319743488 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16379588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1431008355920384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423683087666103 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86596785 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _runtime‚8009.910137861 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1431008355920384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86606762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1421119483961344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1431007997541888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9873427372725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74138378 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86606762 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1421119501000704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9888854919680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74138380 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9888513580544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735163 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74138378 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160745142 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1421119483961344 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423682730076853 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9888513580544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323042736590336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74138380 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20102807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1431007997541888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9888854919680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1421119501000704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9993992272384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.aÖÑvê\llocated‚1413809319743488 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16379588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1431008355920384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423683087666103 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86596785 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _runtime‚8009.910137861 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1431008355920384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86606762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1421119483961344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1431007997541888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9873427372725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74138378 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160761840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74146010 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86605837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9874778976449 % _timestamp‚1.7843483518297591e+09 _runtime‚8010.523535505 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1423830609391809 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1431156251198464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615828 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20104558 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323077292550144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36485891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20104818 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160751847 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1413955813376000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16381333 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74146012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1421266742030336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9889526207488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323077187465216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9874438426447 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1423830251802447 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1431156609584128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36486154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313082259556352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615828 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1431156609584128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9889526207488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1421266742030336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1421266724990976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9889867553792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚769 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9889867553792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313082278168576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1413955830415360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160751847 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74146012o{™{'ðýM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9889867553792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313082278168576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1413955830415360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160751847 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74146012 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74146010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16381336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160761840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74146010 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86605837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9874778976449 % _timestamp‚1.7843483518297591e+09 _runtime‚8010.523535505 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1423830609391809 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1431156251198464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615828 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20104558 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323077292550144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36485891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20104818 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160751847 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1413955813376000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16381333 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74146012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1421266742030336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9889526207488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323077187465216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9874438426447 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1423830251802447 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1431156609584128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36486154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313082259556352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615828 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1431156609584128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9889526207488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1421266742030336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1421266724990976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9889867553792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚769 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269KG%2ñ- _wandbruntime‚8010  _runtime‚8010¿ òƒ 2loss/SRqqt1/assignment_loss‚0.5147165711040173 1loss/SRqqt2/detection_loss‚0.6189611508385514 0loss/FBt2/detection_loss‚0.41168256302122386 % _timestamp‚1.7843483518335245e+09 _runtime‚8010.526824584 /loss/FRt1/detection_loss‚0.5732118716895938 1loss/SRqqt1/detection_loss‚0.8385887309078409 epoch‚23 .loss/FRt1/assignment_loss‚1.47644158753807 0loss/FRt2/assignment_loss‚0.7709733410519171 2loss/FBt1/assignment_loss‚0.002982551686915846 /loss/FRt2/detection_loss‚1.1835331226062278 /loss/FBt1/detection_loss‚0.8235731496764668 %loss/total_loss‚7.675197337495803 _step‚770 3loss/SRqqt2/assignment_loss‚0.44677943155870836 1loss/FBt2/assignment_loss‚0.01375326581626853 trainer/global_step‚9349A¡Ýâóˆ _step‚770 3loss/SRqqt2/assignment_loss‚0.44677943155870836 1loss/FBt2/assignment_loss‚0.01375326581626853 trainer/global_step‚9349 2loss/SRqqt1/assignment_loss‚0.5147165711040173 1loss/SRqqt2/detection_loss‚0.6189611508385514 0loss/FBt2/detection_loss‚0.41168256302122386 % _timestamp‚1.7843483518335245e+09 _runtime‚8010.526824584 /loss/FRt1/detection_loss‚0.5732118716895938 1loss/SRqqt1/detection_loss‚0.8385887309078409 epoch‚23 .loss/FRt1/assignment_loss‚1.47644158753807 0loss/FRt2/assignment_loss‚0.7709733410519171 2loss/FBt1/assignment_loss‚0.002982551686915846 /loss/FRt2/detection_loss‚1.1835331226062278 /loss/FBt1/detection_loss‚0.8235731496764668 %loss/total_loss‚7.675197337495803‚ÖÝ»Q“ô‚(:ˆ ÎýëÒØ÷î“ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áƒð=çõ‚(:Ü ÎýëÒÀ¢Ã• disk.sda1.in‚0 disk.dm-0.in‚14937.62890625 disk.rbd1.out‚ 19.5390625# proc.memory.availableMB‚489.625 cpu‚26.179937282029776$ disk.nvme0c0n1.in‚14937.62890625( disk.nvme0c0n1.out‚25421.94677734375 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚25421.94677734375$ memory_percent‚97.01156616210938* proc.memory.percent‚16.698813438415527# disk.dm-0.out‚25421.94677734375 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 network.sent‚ 1.4916644e+07 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚14937.62890625 disk.rbd1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2735.93359375* disk./.usagePercent‚39.491232233486734$ disk./.usageGB‚588.4478988647461 disk.dm-1.out‚ 255.265625 disk.drbd2165.in‚0 disk.sda3.out‚ 255.265625 disk.sda1.out‚0 disk.sda.out‚ 255.265625 network.recv‚ 2.199001e+06 disk.rbd0.in‚0]C{’ö‚(:‡ ÝýëÒ¥ñ©• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;._!ê÷‚(:ß ÝýëÒåÙ©– disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2735.94921875 cpu‚26.179752425521116 network.recv‚ 2.203282e+06* disk./.usagePercent‚39.491196392431945$ disk./.usageGB‚588.4473648071289 disk.sda2.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚25425.1181640625 disk.rbd1.out‚ 19.6015625# memory_percent‚97.0130443572998 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚15029.05078125" disk.dm-0.out‚25425.1181640625" disk.nvme0n1.in‚15029.05078125 disk.rbd1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 489.3828125* proc.memory.percent‚16.698908805847168 disk.rbd0.in‚0 disk.dm-1.out‚ 255.4921875 disk.drbd2165.out‚0 network.sent‚ 1.4969031e+07 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚15029.05078125 disk.rbd0.out‚0% disk.nvme0n1.out‚25425.1181640625 disk.sda3.out‚ 255.4921875 disk.sda.out‚ 255.4921875 proc.cpu.threads‚17ùÄ$}2ø- _wandbruntime‚8050  _runtime‚8050éæ\…ÇùÁ _step‚771 & lr-AdamW/pg1‚0.0002532875081082486 & lr-AdamW/pg2‚0.0002532875081082486 trainer/global_step‚9399 % _timestamp‚1.7843483921867924e+09 _runtime‚8050.8844596469p?ÞÌúÆ & lr-AdamW/pg1‚0.0002532875081082486 & lr-AdamW/pg2‚0.0002532875081082486 trainer/global_step‚9399 % _timestamp‚1.7843483921867924e+09 _runtime‚8050.884459646 _step‚771ƒ³½72û- _wandbruntime‚8050  _runtime‚8050¦ ½4EüðO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87051237 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1431059237746611 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16466838 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9939209082368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314720287336448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74519978 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1428481552482304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74519978 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324765215732736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1438420744525312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581194 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324765329213440 `MDeviceSt1Ë›±atsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10044947008512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9939550421504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9939550421504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10045041876992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9939209082368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74519980 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8050.887720051 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1428481535442944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87051237 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314720268724224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101ÍŽ3û'ýõO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16466841 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924043859633 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843483921940322e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20205199 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1431058880157361 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚9399 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1438421102903808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87061214 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87051237 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1431059237746611 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16466838 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9939209082368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314720287336448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74519978 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1428481552482304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74519978 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324765215732736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1438420744525312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581194 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324765329213440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10044947008512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9939550421504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9939550421504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10045041876992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9939209082368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74519980 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8050.887720051 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1428481535442944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87051237 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314720268724224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚8051.501453558 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16468586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314753697482752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87070280 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421281364008960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1438569356567552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87060289 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1438569356567552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1428628793511936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74527612 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784348392807623e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87070280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74527610 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36675602 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161597892 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1428628793511936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9940221709312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16468583 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74527610 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36675873 trainer/global_step‚9399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324799658669568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1428628776472576 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421281346969600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74527612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9925395463357 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20207287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1431206759472317 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9940563055616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314753716094976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324799772143104'+nT€þM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74527610 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36675602 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161597892 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1428628793511936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9940221709312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16468583 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74527610 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36675873 trainer/global_step‚9399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324799658669568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1428628776472576 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421281346969600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74527612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9925395463357 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20207287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.alloca‘íåYted‚1431206759472317 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9940563055616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314753716094976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324799772143104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1428628776472576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚8051.501453558 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16468586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314753697482752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87070280 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421281364008960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1438569356567552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87060289 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1438569356567552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1428628793511936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74527612 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784348392807623e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87070280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1445833491508736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74901580 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74901580 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1435843586924544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326487608217088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162417246 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162407267 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1445833849887232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1435843603963904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316391725262848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36861863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10096000629248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20307775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360·¼äû'õO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162417246 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162407267 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1445833849887232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1435843603963904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316391725262848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36861863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10096000629248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20307775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1435843586924544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87515666 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9990245923328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 _runtime‚8091.882705049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36862136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326487725892096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87505689 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1445833849887232 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1445833491508736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1428460386930688 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87515666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1438435387827119 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9989904584192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1445833491508736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74901580 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74901580 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1435843586924544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326487608217088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ˆÖÕn»2- _wandbruntime‚8092  _runtime‚8092 èÝ_†‘ùM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocaÆùnêy&tion.all.freed‚ 162423951 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843484338028705e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74909212 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975671400263 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1438582909552825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74909212 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚777 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36865630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162423951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16555836 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9991258557440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433944 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87524732 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36865893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16555833 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10096946954240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1435990827954176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E7²'’þM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9991258557440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9990917211136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20310057 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87514741 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36865630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162423951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16555836 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9991258557440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433944 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87524732 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36865893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16555833 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10096946954240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1435990827954176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162423951 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843484338028705e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74909212 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975671400263 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1438582909552825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74909212 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚777 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20410380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1445811537907627 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87960141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435785920524288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75283180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318063163189248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1453246238492160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_byt_=­šes.small_pool.freed‚0 trainer/global_step‚9499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1453246596870656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1443205655445504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87970118 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _step‚780 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75283178 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1443205638406144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163243319 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J(¡ûO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1443205638406144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16641338 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1453246238492160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16641341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1453246238492160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1453246596870656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1443205655445504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87970118 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _step‚780 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75283178 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1443205638406144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163243319 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20410380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1445811537907627 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87960141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435785920524288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75283180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318063163189248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10041954059264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843484745876725e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87979184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37055786 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37055512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328244582523904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87969193 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1443352896475136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318096573335552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1453394492148736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75290810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16643086 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1453394850534400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20412700 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163269996 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10148009188352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10026628437173 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1443352879435776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1445959059633333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87979184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20412429 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10041954059264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843484745876725e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87979184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37055786 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37055512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328244582523904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87969193 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1443352896475136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318096573335552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1453394492148736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75290810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16643086 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1453394850534400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20412700 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163269996 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10148009188352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10026628437173 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1443352879435776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1445959059633333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87979184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20412429 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37242092 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1460658985475584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10198158929920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75664780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329932652856320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚8173.759458784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10198070352896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88424570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1453187687988135 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88414593 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75664778 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10076233870247 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10091636926976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20513227 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88414593 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1460659343854080 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37241815 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1443111454117888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16728591 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1453187330398885 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1450567706927104 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20513501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843485150657203e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10091295587840 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1450567689887744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1460659343854080©`8(±ûO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88424570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16728588 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10075893320357 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚784 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37242092 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1460658985475584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10198158929920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75664780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329932652856320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚8173.759458784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10198070352896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88424570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1453187687988135 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88414593 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75664778 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10076233870247 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10091636926976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20513227 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88414593 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1460659343854080 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37241815 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1443111454117888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16728591 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1453187330398885 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1450567706927104 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20513501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843485150657203e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10091295587840 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1450567689887744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1460659343854080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164089350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1450567706927104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1443111437078528 trainer/global_step‚9549#$G¢2²- _wandbruntime‚8174  _runtime‚8174ðéqÇ<³€N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1453334852124479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10199181696512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164106048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.largÌÈ¢Êe_pool.freed‚16730333 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450714947956736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37245648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1443257947750400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37245909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16730336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚9549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1460807597517824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚785 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75672410 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450714930917376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329967211570688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88433636 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88423645 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843485156837568e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1460807597517824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096055 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460807239132160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10092649561088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976"axu '´…N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88423645 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88433636 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20515573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450714930917376 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1453334852124479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10199181696512 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164106048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16730333 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450714947956736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37245648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1443257947750400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37245909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16730336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚9549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1460807597517824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚785 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75672410 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450714930917376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329967211570688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88433636 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88423645 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843485156837568e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1460807597517824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096055 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460807239132160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10092649561088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 Qþžº‚(:“ óþëÒϯæ“ gpu.0.gpu‚60.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù‹¸ãÞ»‚(:Ó óþëÒΪ”— disk.nvme0n1.in‚ 15317.125$ disk./.usageGB‚588.4479637145996 disk.rbd1.out‚ 19.9765625 disk.rbd0.in‚0$ memory_percent‚97.01321125030518! proc.memory.rssMB‚ 2736.078125 disk.nvme0c0n1.in‚ 15317.125 cpu‚26.17673995071271& disk.nvme0n1.out‚25464.83544921875 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 259.01953125 disk.dm-0.in‚ 15317.125 network.sent‚ 1.5250198e+07 network.recv‚ 2.24189e+06) disk./.usagePercent‚39.49123658561482 disk.drbd2165.in‚0# disk.dm-0.out‚25464.83544921875( proc.memory.availableMB‚ 489.35546875 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 259.01953125 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25464.83544921875 disk.sda.in‚ 0.1328125 disk.sda.out‚ 259.01953125* proc.memory.percent‚16.699695587158203 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0–%rž¼‚(:“ ‚ÿëÒ¼ÉÌ“ gpu.0.gpu‚47.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚151.793' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.358600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇKaOͽ‚(: ‚ÿëÒÚöו disk.rbd0.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚25468.2275390625" disk.dm-0.out‚25468.2275390625 disk.rbd1.in‚0* proc.memory.percent‚16.699695587158203) disk./.usagePercent‚39.49123453755454 disk.nvme0c0n1.in‚ 15317.125 disk.dm-0.in‚ 15317.125 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚25468.2275390625 disk.sda3.out‚259.375 disk.rbd1.out‚ 19.9765625 network.recv‚ 2.244935e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 488.84765625 network.sent‚ 1.5255188e+07 disk.sda2.in‚0! proc.memory.rssMB‚ 2736.078125 disk.sda.out‚259.375 disk.sda2.out‚0# memory_percent‚97.0163106918335 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚259.375 disk.nvme0n1.in‚ 15317.125 cpu‚26.176275875875465$ disk./.usageGB‚588.4479331970215 disk.dm-1.in‚ 0.1328125 disk.sda3.in‚ 0.1328125ëN È2¾- _wandbruntime‚8214  _runtime‚8214ö̹7È¿Â $ _timestamp‚1.784348555984777e+09 _runtime‚8214.680499347 _step‚787 ' lr-AdamW/pg1‚0.00024613515841978144 ' lr-AdamW/pg2‚0.00024613515841978144 trainer/global_step‚9599 CðlÍÀÇ ' lr-AdamW/pg2‚0.00024613515841978144 trainer/global_step‚9599 $ _timestamp‚1.784348555984777e+09 _runtime‚8214.680499347 _step‚787 ' lr-AdamW/pg1‚0.00024613515841978144“ÿûZ2Á- _wandbruntime‚8214  _runtime‚8214^ …û'ÂõO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37431946 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10249192033280 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1457929741369344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10141991089664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚8214.684460585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76046380 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76046378 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37431682 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10249099261952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10126850357155 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1450436970672128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88879022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16815838 _step‚788 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76046378 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10142332428800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1457929758408704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1450436987711488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20615844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331655119691776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10126509807265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468071732459008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16815841 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142332428800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88879022 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88869045 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468072090837504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1460563838068643 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1457929741369344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784348555987737e+09ÒúŽ ÃúO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1450436970672128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88879022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16815838 _step‚788 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76046378 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10142332428800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1457929758408704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1450436987711488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20615844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331655119691776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10126509807265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_tr;µ¯èàain_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468071732459008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16815841 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142332428800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88879022 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88869045 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468072090837504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1460563838068643 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1457929741369344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784348555987737e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37431946 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10249192033280 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1457929741369344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10141991089664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚8214.684460585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76046380 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76046378 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37431682 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10249099261952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10126850357155 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250”³i‘Ã2Ä- _wandbruntime‚8215  _runtime‚8215­ÁO'Å€N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10127520860987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164932107 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1460711359794349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1468219986115584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468220344501248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942100 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88878097 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1458076999438336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331689679989248 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10127861410989 _runtime‚8215.299277976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76054012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331689570710016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10250212188672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76054012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88888088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843485566054547e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1460711002204987 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321439467800576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468220344501248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚9599 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88878097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20617874 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37435457 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1450583498383360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37435734 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143345062912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16817586 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚789 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1458076982398976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20618148 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143345062912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1458076999438336 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16817583 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321439449188352 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1450583481344000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0mì$K 'Æ…N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20617874 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37435457 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1450583498383360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37435734 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143345062912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16817586 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚789 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1458076982398976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20618148 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143345062912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1458076999438336 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16817583 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321439449188352 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1450583481344000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10127520860987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164932107 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1460711359794349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1468219986115584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468220344501248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942100 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88878097 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1458076999438336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331689679989248 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10127861410989 _runtime‚8215.299277976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76054012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331689570710016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10250212188672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76054012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88888088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843485566054547e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1460711002204987 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321439467800576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1468220344501248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚9599 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88878097•Ç‰2Ç- _wandbruntime‚8215  _runtime‚82150e!g ȃ _step‚790 0loss/FRt2/assignment_loss‚0.7939846814910242 2loss/SRqqt2/assignment_loss‚0.4210695260475267 /loss/FRt1/detection_loss‚0.5897395084327711 trainer/global_step‚9599 _runtime‚8215.301598056 1loss/SRqqt1/detection_loss‚0.8704815427188799 .loss/FBt1/detection_loss‚0.577263666089715 %loss/total_loss‚7.933691692214157 $ _timestamp‚1.784348556608274e+09 2loss/SRqqt1/assignment_loss‚0.5122978708461081 2loss/FBt2/assignment_loss‚0.009164600247468275 /loss/FRt2/detection_loss‚1.2321251233164625 1loss/SRqqt2/detection_loss‚0.6784140251184424 /loss/FBt2/detection_loss‚0.6609475426728046 0loss/FRt1/assignment_loss‚1.5871456553800594 3loss/FBt1/assignment_loss‚0.0010579498528935144 epoch‚235YɈ /loss/FBt2/detection_loss‚0.6609475426728046 0loss/FRt1/assignment_loss‚1.5871456553800594 3loss/FBt1/assignment_loss‚0.0010579498528935144 epoch‚23 _step‚790 0loss/FRt2/assignment_loss‚0.7939846814910242 2loss/SRqqt2/assignment_loss‚0.4210695260475267 /loss/FRt1/detection_loss‚0.5897395084327711 trainer/global_step‚9599 _runtime‚8215.301598056 1loss/SRqqt1/detection_loss‚0.8704815427188799 .loss/FBt1/detection_loss‚0.577263666089715 %loss/total_loss‚7.933691692214157 $ _timestamp‚1.784348556608274e+09 2loss/SRqqt1/assignment_loss‚0.5122978708461081 2loss/FBt2/assignment_loss‚0.009164600247468275 /loss/FRt2/detection_loss‚1.2321251233164625 1loss/SRqqt2/detection_loss‚0.6784140251184424–%9ZEœÊ‚(:‘ ‘ÿëÒŽü–” gpu.0.gpu‚31.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+èã½áË‚(:Ö ‘ÿëÒÖö• network.recv‚ 2.248516e+06 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 259.8203125# proc.memory.rssMB‚ 2739.94921875* proc.memory.percent‚16.723322868347168 network.sent‚ 1.5260156e+07 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚15402.24609375 disk.rbd1.in‚0) disk./.usagePercent‚39.49123402553948( disk.nvme0c0n1.out‚25471.00537109375# disk.dm-0.out‚25471.00537109375 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚25471.00537109375 disk.rbd1.out‚ 20.0390625 proc.memory.availableMB‚485 disk.sda.in‚ 0.1328125 disk.sda.out‚ 259.8203125 disk.drbd2165.in‚0 disk.dm-0.in‚15402.24609375 disk.rbd0.in‚0 cpu‚26.17734782963618 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 259.8203125 disk.rbd0.out‚0" disk.nvme0n1.in‚15402.24609375# disk./.usageGB‚588.447925567627" memory_percent‚97.039794921875 proc.cpu.threads‚17²_ÄÙ2Ì- _wandbruntime‚8233  _runtime‚8233g¯«@˜Íµ{ 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2025717794895172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _runtime‚8233.877303834 _step‚791 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888898730278015 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6480411291122437 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03192529082298279 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7769374847412109 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.686220645904541 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6480207443237305 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708134651184 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30401426553726196 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2F\$#8FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.875472366809845 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.500525176525116 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8606059551239014 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.913099467754364 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6239491701126099 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7535713315010071 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6329529285430908 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8606059551239014 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3781278431415558 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8754912614822388 +jet/accuracy_2_of_2‚0.15201668441295624 particle/accuracy_3_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09694693982601166 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8423316478729248 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2733934223651886 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048084646463394165 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26909753680229187 6"Purity/*FRt*SRqqt2FBt/event_purity‚0.8890380859375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5132192969322205 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.054915934801101685 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963852763175964 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5282705426216125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7810322046279907 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22498537600040436 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11511147767305374 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02757318504154682 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7195063829421997 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02875404618680477 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3162209689617157 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6validation_average_jet_accuracy‚0.3740915060043335 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956672191619873 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26908257603645325 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03228973597288132 particle/accuracy_1_of_3‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_4‚0.7625046968460083 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6638447642326355 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37841036915779114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5788219571113586 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6651226878166199 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.35806310176849365 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22050249576568604 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9002078175544739 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +particle/specificity‚0.9061055183410645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899484515190125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6378601789474487 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8171013593673706 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7095633745193481 +jet/accuracy_3_of_3‚0.06568751484155655 trainer/global_step‚9599 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5749437808990479 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9510167241096497 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.07521244138479233 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26075759530067444 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20536106824874878 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23724347352981567 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01630210503935814 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 epoch‚23 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957027435302734 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22596797347068787 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8942553997039795 +jet/accuracy_4_of_4‚0.01670061983168125 )loss/val_total_loss‚7.710199349628505 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6691527366638184 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5616666078567505 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6910446286201477 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3385315537452698 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2253505140542984 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6844663619995117 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.037388306111097336 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6998084187507629 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7187828421592712 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4920295178890228 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882912635803223 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948529005050659 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19865138828754425 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7158109545707703 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4515201449394226 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13326357305049896 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027357008308172226 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24247705936431885 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 particle/accuracy_2_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996489942073822 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9194262623786926 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9759686589241028 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988171458244324 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6294844746589661 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2618207633495331 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24810735881328583 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11488628387451172 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6507954597473145 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8577818274497986 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4962249994277954 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_2‚0.7545103430747986 /particle/accuracy_3_of_4‚0.9944444298744202 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9042705297470093 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9879844188690186 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7178261876106262 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.742803156375885 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3363932967185974 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25294262170791626 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7811077833175659 +particle/sensitivity‚0.6091368794441223 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.387901246547699 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9166989922523499 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21244893968105316 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7598884701728821 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039737552404403687 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07167786359786987 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2346462458372116 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6598559617996216 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7010459899902344 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7716702222824097 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7131252884864807 *jet/accuracy_2_of_3‚0.5537815690040588 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6653143167495728 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6529433727264404 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6264714598655701 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21366626024246216 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01630210503935814 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23028872907161713 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27309155464172363 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8485532402992249 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9781327247619629 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6383076906204224 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7113732099533081 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6383076906204224 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7017892599105835 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843485751830366e+09 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2618207633495331éZñÀ=κ{ 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7535713315010071 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6329529285430908 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8606059551239014 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3781278431415558 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8754912614822388 +jet/accuracy_2_of_2‚0.15201668441295624 particle/accuracy_3_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09694693982601166 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8423316478729248 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2733934223651886 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048084646463394165 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26909753680229187 6"Purity/*FRt*SRqqt2FBt/event_purity‚0.8890380859375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5132192969322205 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.054915934801101685 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963852763175964 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5282705426216125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7810322046279907 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22498537600040436 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11511147767305374 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02757318504154682 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7195063829421997 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02875404618680477 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3162209689617157 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6validation_average_jet_accuracy‚0.3740915060043335 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956672191619873 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26908257603645325 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03228973597288132 particle/accuracy_1_of_3‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_4‚0.7625046968460083 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6638447642326355 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37841036915779114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5788219571113586 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6651226878166199 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.35806310176849365 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22050249576568604 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9002078175544739 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +particle/specificity‚0.9061055183410645 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899484515190125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6378601789474487 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8171013593673706 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7095633745193481 +jet/accuracy_3_of_3‚0.06568751484155655 trainer/global_step‚9599 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5749437808990479 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9510167241096497 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.07521244138479233 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26075759530067444 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20536106824874878 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23724347352981567 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01630210503935814 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 epoch‚23 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9957027435302734 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22596797347068787 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8942553997039795 +jet/accuracy_4_of_4‚0.01670061983168125 )loss/val_total_loss‚7.710199349628505 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6691527366638184 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5616666078567505 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6910446286201477 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3385315537452698 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2253505140542984 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6844663619995117 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.037388306111097336 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6998084187507629 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7187828421592712 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4920295178890228 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882912635803223 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9948529005050659 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19865138828754425 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7158109545707703 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4515201449394226 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13326357305049896 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027357008308172226 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24247705936431885 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 particle/accuracy_2_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996489942073822 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9194262623786926 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9759686589241028 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988171458244324 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6294844746589661 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2618207633495331 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24810735881328583 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11488628387451172 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6507954597473145 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8577818274497986 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4962249994277954 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_2‚0.7545103430747986 /particle/accuracy_3_of_4‚0.9944444298744202 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9042705297470093 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9879844188690186 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7178261876106262 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.742803156375885 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3363932967185974 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25294262170791626 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7811077833175659 +particle/sensitivity‚0.6091368794441223 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.387901246547699 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9166989922523499 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21244893968105316 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7598884701728821 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039737552404403687 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07167786359786987 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2346462458372116 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6598559617996216 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7010459899902344 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7716702222824097 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7131252884864807 *jet/accuracy_2_of_3‚0.5537815690040588 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6653143167495728 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6529433727264404 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6264714598655701 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21366626024246216 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01630210503935814 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23028872907161713 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27309155464172363 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8485532402992249 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9781327247619629 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6383076906204224 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7113732099533081 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6383076906204224 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7017892599105835 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843485751830366e+09 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2618207633495331 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2025717794895172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _runtime‚8233.877303834 _step‚791 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888898730278015 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6480411291122437 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03192529082298279 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7769374847412109 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.686220645904541 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6480207443237305 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708134651184 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30401426553726196 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.875472366809845 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.500525176525116 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8606059551239014 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.913099467754364 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6239491701126099—ö+&Y‘Ï‚(:†  ÿëÒˆ³Ÿ• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}L¼¨ïЂ(:ä  ÿëÒÃÙ– disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125$ memory_percent‚97.03974723815918* disk./.usagePercent‚39.491233513524406 disk.drbd2165.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚25475.43310546875 disk.rbd1.in‚0 disk.dm-1.out‚ 259.96484375( disk.nvme0c0n1.out‚25475.43310546875 disk.dm-0.in‚15402.24609375" disk.nvme0n1.in‚15402.24609375# proc.memory.rssMB‚ 2739.94921875 proc.cpu.threads‚17$ disk./.usageGB‚588.4479179382324 disk.sda3.out‚ 259.96484375 disk.rbd1.out‚ 20.0390625 network.recv‚ 2.25273e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 259.96484375 network.sent‚ 1.532312e+07 disk.sda1.out‚0# disk.dm-0.out‚25475.43310546875 disk.sda2.out‚0' proc.memory.availableMB‚ 485.0078125* proc.memory.percent‚16.723322868347168 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚15402.24609375 cpu‚26.181679736606114ë>`“Ñ‚(:ˆ ¯ÿëÒþ°÷“ gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚195.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.1676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ Â&àÒ‚(:Ý ¯ÿëÒè˜ö–! proc.memory.rssMB‚ 2739.984375) proc.memory.percent‚16.72353744506836$ disk./.usageGB‚588.4478569030762% disk.nvme0n1.out‚25478.1494140625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 network.sent‚ 1.5380475e+07 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.03922271728516 proc.cpu.threads‚17" disk.nvme0n1.in‚15402.24609375) disk./.usagePercent‚39.49122941740386" disk.dm-0.out‚25478.1494140625 disk.sda2.out‚0 disk.rbd1.out‚ 20.1640625$ disk.nvme0c0n1.in‚15402.24609375 disk.sda.out‚ 260.25390625 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125% proc.memory.availableMB‚ 485.09375 cpu‚26.181228339476593 network.recv‚Ç4‘S 2.257027e+06 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 260.25390625 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚25478.1494140625 disk.dm-0.in‚15402.24609375 disk.sda3.out‚ 260.25390625’böžÓ‚(:“ ¾ÿëÒÃûê“ gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì>ÀæéÔ‚(:Þ ¾ÿëÒÞК— disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda3.in‚ 0.1328125$ memory_percent‚97.03946113586426' proc.memory.availableMB‚ 485.0546875$ disk./.usageGB‚588.4479904174805 disk.rbd0.out‚0) proc.memory.percent‚16.72353744506836) disk./.usagePercent‚39.49123837766756 disk.dm-1.out‚ 260.50390625 disk.drbd2165.out‚0" disk.nvme0n1.in‚15402.24609375 cpu‚26.18101921326887" disk.dm-0.out‚25482.3818359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 20.1640625 proc.cpu.threads‚17 disk.sda1.in‚0$ disk.nvme0c0n1.in‚15402.24609375 disk.dm-0.in‚15402.24609375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 260.50390625! proc.memory.rssMB‚ 2739.984375 network.sent‚ 1.5385448e+07 network.recv‚ 2.260079e+06 disk.rbd0.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚25482.3818359375' disk.nvme0c0n1.out‚25482.3818359375 disk.sda3.out‚ 260.50390625M3qf2Õ- _wandbruntime‚8274  _runtime‚8274ë=–ÙÉÖà _runtime‚8274.068470132 _step‚792 ' lr-AdamW/pg1‚0.00024434331583894696 ' lr-AdamW/pg2‚0.00024434331583894696 trainer/global_step‚9649 % _timestamp‚1.7843486153717883e+09qåB©Î×È trainer/global_step‚9649 % _timestamp‚1.7843486153717883e+09 _runtime‚8274.068470132 _step‚792 ' lr-AdamW/pg1‚0.00024434331583894696 ' lr-AdamW/pg2‚0.00024434331583894696˜ÏÇ=2Ø- _wandbruntime‚8274  _runtime‚8274‘E4«û'ÙõO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1477431779298304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37655534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89393819 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16905608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333411819642368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76548646 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89403796 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323093370456064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16905611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1477432137676800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10210419796992 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1459671184519168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195186569292 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10318449186304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843486153757486e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1467221359501312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20750188 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469866371088460 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20749926 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1459671167479808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165952442 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1467221376540672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚9649 _runtime‚8274.069423131 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1467221359501312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194846019402 _step‚793 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1477431779298304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323093389068288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10210419796992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37655799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942463 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1467221376540672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚9649 _runtime‚8274.069423131 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1467221359501312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194846019402 _step‚793 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1477431779298304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323093389068288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10210419796992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37655799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942463 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1477431779298304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37655534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89393819 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16905608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333411819642368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76548646 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89403796 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323093370456064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16905611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1477432137676800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10210419796992 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1459671184519168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195186569292 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10318449186304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843486153757486e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1467221359501312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20750188 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469866371088460 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0™»°¤R2Û- _wandbruntime‚8274  _runtime‚8274»³æü&Ü€N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1477580391340544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10211773770240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76556278 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37659571 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1467368600530944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323126817826816 trainer/global_step‚9649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333446374680576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10211773770240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16907353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10211432423936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20751950 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89402871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89402871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20752215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1467368617570304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333446271692800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165959147 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319556853760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959147 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1467368600530944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89412862 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1467368617570304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323126799214592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10195857073124 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969140 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76556278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319472478208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76556276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89412862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1477580032954880 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1477580391340544 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37659303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚8274DS½U .683974877¥8` 'Ý…N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89402871 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89402871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20752215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1467368617570304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333446271692800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165959147 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319556853760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959147 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1467368600530944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89412862 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1467368617570304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323126799214592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10195857073124 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969140 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76556278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319472478208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76556276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89412862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1477580032954880 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1477580391340544 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37659303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚8274.683974877 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1477580391340544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10211773770240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76556278 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37659571 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1467368600530944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323126817826816 trainer/global_step‚9649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333446374680576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10211773770240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16907353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10211432423936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20751950š')2Þ- _wandbruntime‚8274  _runtime‚8274ƒjû ߃ trainer/global_step‚9649 _runtime‚8274.687676745 _step‚795 0loss/FRt2/assignment_loss‚0.9262068982855732 /loss/FRt2/detection_loss‚1.2253595352864155 1loss/SRqqt1/detection_loss‚0.9186266435222721 epoch‚24 $ _timestamp‚1.784348615994369e+09 3loss/SRqqt2/assignment_loss‚0.45256724163720075 2loss/FBt1/assignment_loss‚0.011619265840999012 /loss/FRt1/detection_loss‚0.5296329389384966 /loss/FBt1/detection_loss‚0.5470543271889362 2loss/FBt2/assignment_loss‚0.009606413345001024 /loss/FBt2/detection_loss‚0.5080217265253925 0loss/FRt1/assignment_loss‚1.5366900136012485 1loss/SRqqt1/assignment_loss‚0.469278201506884 1loss/SRqqt2/detection_loss‚0.6032465838923083 %loss/total_loss‚7.737909789570727ÙÞàˆ 2loss/FBt1/assignment_loss‚0.011619265840999012 /loss/FRt1/detection_loss‚0.5296329389384966 /loss/FBt1/detection_loss‚0.5470543271889362 2loss/FBt2/assignment_loss‚0.009606413345001024 /loss/FBt2/detection_loss‚0.5080217265253925 0loss/FRt1/assignment_loss‚1.5366900136012485 1loss/SRqqt1/assignment_loss‚0.469278201506884 1loss/SRqqt2/detection_loss‚0.6032465838923083 %loss/total_loss‚7.737909789570727 trainer/global_step‚9649 _runtime‚8274.687676745 _step‚795 0loss/FRt2/assignment_loss‚0.9262068982855732 /loss/FRt2/detection_loss‚1.2253595352864155 1loss/SRqqt1/detection_loss‚0.9186266435222721 epoch‚24 $ _timestamp‚1.784348615994369e+09 3loss/SRqqt2/assignment_loss‚0.45256724163720075›w#I“á‚(:ˆ Íÿë҆ǔ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Y0?äâ‚(:Ù ÍÿëÒÀð“—$ disk./.usageGB‚588.4478530883789 disk.dm-1.in‚ 0.1328125" proc.memory.rssMB‚ 2739.9921875 network.recv‚ 2.263688e+06 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125$ memory_percent‚97.04105854034424 cpu‚26.181053815758148 disk.dm-0.in‚15452.26171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 25485.3203125 disk.dm-0.out‚ 25485.3203125" disk.nvme0n1.in‚15452.26171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 484.79296875* disk./.usagePercent‚39.491229161396326$ disk.nvme0c0n1.out‚ 25485.3203125 disk.sda1.out‚0 network.sent‚ 1.5390461e+07 disk.rbd1.out‚ 20.2265625 disk.sda1.in‚0 disk.dm-1.out‚ 260.86328125) proc.memory.percent‚16.72358512878418$ disk.nvme0c0n1.in‚15452.26171875 disk.sda.out‚ 260.86328125 disk.sda3.out‚ 260.86328125È) ã‚(:’ ÜÿëÒí‘š” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.69' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.337999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ó°«âä‚(:× ÜÿëÒ—þô• disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.sda2.in‚0" disk.nvme0n1.in‚15452.26171875 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 20.2265625 proc.memory.rssMB‚2740 network.recv‚ 2.267837e+06) disk./.usagePercent‚39.49098416218605 disk.drbd2165.in‚0& disk.nvme0n1.out‚25489.78076171875 disk.rbd1.in‚0% proc.memory.percent‚ 16.7236328125 cpu‚26.180705648721815$ disk./.usageGB‚588.4442024230957 disk.sda.out‚ 261.19921875' proc.memory.availableMB‚ 484.4921875 disk.dm-0.in‚15452.26171875 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15452.26171875 disk.rbd0.in‚0 disk.dm-1.out‚ 261.19921875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚97.04289436340332 network.sent‚ 1.5453464e+07 disk.rbd0.out‚0( disk.nvme0c0n1.out‚25489.78076171875# disk.dm-0.out‚25489.78076171875 disk.sda3.out‚ 261.19921875uÕ^žå‚(:“ ëÿëÒ¿šÁ“ gpu.0.gpu‚69.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚181.517' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.303399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%(,&Ùæ‚(:Î ëÿëÒÌ‹ý– network.recv‚ 2.271793e+06 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.dm-1.out‚ 261.49609375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚25494.71484375$ memory_percent‚97.04041481018066$ disk./.usageGB‚588.4442558288574# disk.nvme0n1.out‚25494.71484375' proc.memory.availableMB‚ 484.8984375 disk.drbd2165.out‚0 disk.dm-0.in‚15452.26171875" disk.nvme0n1.in‚15452.26171875 disk.dm-1.in‚ 0.1328125 disk.dm-0.out‚25494.71484375 disk.sda.out‚ 261.49609375% proc.memory.percent‚ 16.7236328125 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15452.26171875 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 261.49609375 network.sent‚ 1.5458681e+07) disk./.usagePercent‚39.49098774629153 disk.rbd1.out‚ 20.2890625 proc.memory.rssMB‚2740 cpu‚26.180253731653124‹®ìu2ç- _wandbruntime‚8315  _runtime‚8315WœðKÉèà trainer/global_step‚9699 % _timestamp‚1.7843486563187678e+09 _runtime‚8315.016498159 _step‚796 ' lr-AdamW/pg1‚0.00024255020122995795 ' lr-AdamW/pg2‚0.00024255020122995795%ÌüÎéÈ % _timestamp‚1.7843486563187678e+09 _runtime‚8315.016498159 _step‚796 ' lr-AdamW/pg1‚0.00024255020122995795 ' lr-AdamW/pg2‚0.00024255020122995795 trainer/global_step‚9699œ Z  2ê- _wandbruntime‚8315  _runtime‚8315Ã=õ;û'ëõO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76930244 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1474583410982912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89858248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1474583410982912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10261456637952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166778515 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10261115298816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16992858 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1477242521168968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1484844884660224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1484844526281728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37846630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89848271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1477242163579718 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1474583428022272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89858248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76930244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10369887249408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10245462506310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16992861 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335134800723968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20854044 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 trainer/global_step‚9699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1466996718112768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10245803056200 _runtime‚8315.020985199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324764826994688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚797 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76930246 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10261456637952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788494 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20853772 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335134695631872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1484844526281728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37846630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89848271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1477242163579718 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1474583428022272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89858248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76930244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10369887249408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10245462506310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16992861 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335134800723968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20854044 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 trainer/global_step‚9699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStagª~8tsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1466996718112768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10245803056200 _runtime‚8315.020985199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324764826994688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚797 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76930246 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10261456637952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788494 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20853772 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335134695631872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166778515 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10261115298816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16992858 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1477242521168968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1484844884660224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37850479 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10262469272064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16994603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1467143211745280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10246473560032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335169251694080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚9699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89857323 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843486569407165e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1484993138323968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324798237140992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1474730669051904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1474730652012544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10262127925760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1477390042894674 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335169146609152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166795199 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1474730652012544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37850753 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10262127925760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1467143228784640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76937876 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚798 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1474730669051904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89867314 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1484992779938304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10370909468160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324798255753216 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166795199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89857323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16994606 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10370995940864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3¤·P& 'ï…N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166795199 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1474730652012544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37850753 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10262127925760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1467143228784640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76937876 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚798 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1474730669051904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89867314 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1484992779938304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10370909468160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324798255753216 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166795199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89857323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16994606 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10370995940864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37850479 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10262469272064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16994603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1467143211745280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10246473560032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335169251694080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚9699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89857323 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843486569407165e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1484993138323968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324798237140992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1474730669051904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1474730652012544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10262127925760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1477390042894674 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335169146609152ž ³9÷2ð- _wandbruntime‚8315  _runtime‚8315Ь}ñ‚ /loss/FBt2/detection_loss‚0.4632401208429507 epoch‚24 0loss/FRt2/assignment_loss‚0.8479825674754674 /loss/FRt2/detection_loss‚1.3087989850670114 1loss/SRqqt2/detection_loss‚0.6084556421923079 _runtime‚8315.637644458 2loss/SRqqt2/assignment_loss‚0.5043098317709245 trainer/global_step‚9699 $ _timestamp‚1.784348656944342e+09 %loss/total_loss‚7.657078641986354 0loss/FRt1/assignment_loss‚1.5872787682467668 2loss/FBt1/assignment_loss‚0.016278798881896508 2loss/FBt2/assignment_loss‚0.011881004509404328 /loss/FRt1/detection_loss‚0.5524827909648843 /loss/FBt1/detection_loss‚0.2812856462233784 _step‚799 1loss/SRqqt1/assignment_loss‚0.557035487271057 1loss/SRqqt1/detection_loss‚0.9180489985403056òÀyt ò‡ 1loss/SRqqt1/assignment_loss‚0.557035487271057 1loss/SRqqt1/detection_loss‚0.9180489985403056 /loss/FBt2/detection_loss‚0.4632401208429507 epoch‚24 0loss/FRt2/assignment_loss‚0.8479825674754674 /loss/FRt2/detection_loss‚1.3087989850670114 1loss/SRqqt2/detection_loss‚0.6084556421923079 _runtime‚8315.637644458 2loss/SRqqt2/assignment_loss‚0.5043098317709245 trainer/global_step‚9699 $ _timestamp‚1.784348656944342e+09 %loss/total_loss‚7.657078641986354 0loss/FRt1/assignment_loss‚1.5872787682467668 2loss/FBt1/assignment_loss‚0.016278798881896508 2loss/FBt2/assignment_loss‚0.011881004509404328 /loss/FRt1/detection_loss‚0.5524827909648843 /loss/FBt1/detection_loss‚0.2812856462233784 _step‚799Ÿ=[Løó‚(:’ úÿëÒ· ™” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nnûàô‚(:Õ úÿëÒ«‚’– disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚15452.26171875 disk.sda.in‚ 0.1328125 disk.sda.out‚ 261.953125 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 484.6171875! proc.memory.rssMB‚ 2740.015625) proc.memory.percent‚16.72372817993164$ disk.nvme0n1.out‚25502.634765625 cpu‚26.179965434821845 proc.cpu.threads‚17 network.sent‚ 1.5463547e+07) disk./.usagePercent‚39.49124554587852 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚15452.26171875 disk.rbd1.out‚ 20.2890625 disk.dm-1.out‚ 261.953125" disk.nvme0n1.in‚15452.26171875 network.recv‚ 2.275183e+06& disk.nvme0c0n1.out‚25502.634765625# memory_percent‚97.0421314239502 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚588.4480972290039 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚25502.634765625 disk.sda3.out‚ 261.953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0$aEAõ‚(:’ ‰€ìÒ¸¸Ç“ gpu.0.gpu‚79.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wïG|äö‚(:Ù ‰€ìÒ¿ò÷• disk.drbd2165.out‚0# disk.dm-0.out‚25506.32177734375& proc.memory.availableMB‚ 484.609375 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚25506.32177734375$ memory_percent‚97.04217910766602" proc.memory.rssMB‚ 2740.0234375 disk.sda3.out‚ 262.265625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 15510.6484375 network.recv‚ 2.279801e+06 disk.sda1.out‚0 disk.dm-1.out‚ 262.265625) proc.memory.percent‚16.72377586364746) disk./.usagePercent‚39.49124528987098 disk.rbd1.out‚ 20.3515625 cpu‚26.179733839969952 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 15510.6484375 disk.sda.out‚ 262.265625 network.sent‚ 1.5526576e+07 disk.sda3.in‚ 0.1328125( disk.nvme0c0n1.out‚25506.32177734375 disk.sda2.in‚0$ disk./.usageGB‚588.4480934143066 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 15510.6484375¿Xž “÷‚(:ˆ ˜€ìÒÔà™“ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂKóBêø‚(:ß ˜€ìÒ™ó–# disk.dm-0.out‚25511.75537109375& disk.nvme0n1.out‚25511.75537109375 disk.sda3.out‚ 262.49609375 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 2.283245e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 262.49609375& proc.memory.availableMB‚ 484.609375" proc.memory.rssMB‚ 2740.0234375 network.sent‚ 1.5531545e+07 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 20.3515625 disk.rbd0.in‚0$ memory_percent‚97.04217910766602 proc.cpu.threads‚17 disk.drbd2165.out‚0) proc.memory.percent‚16.72377586364746 cpu‚26.179376182600368# disk.nvme0c0n1.in‚ 15510.6484375 disk.dm-0.in‚ 15510.6484375 disk.sda.out‚ 262.49609375 disk.sda1.out‚0 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 15510.6484375) disk./.usagePercent‚39.49123709762989$ disk./.usageGB‚588.4479713439941( disk.nvme0c0n1.out‚25511.75537109375 disk.sda1.in‚0 disk.drbd2165.in‚0¿0¯2ù- _wandbruntime‚8355  _runtime‚8355Âjž‡ÇúÁ ' lr-AdamW/pg1‚0.00024075592976121517 ' lr-AdamW/pg2‚0.00024075592976121517 trainer/global_step‚9749 $ _timestamp‚1.784348697303791e+09 _runtime‚ 8355.99741897 _step‚800âÕÛaÌûÆ ' lr-AdamW/pg1‚0.00024075592976121517 ' lr-AdamW/pg2‚0.00024075592976121517 trainer/global_step‚9749 $ _timestamp‚1.784348697303791e+09 _runtime‚ 8355.99741897 _step‚800 #©¸p2ü- _wandbruntime‚8355  _runtime‚8355ånBNýõO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1492257273265152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326436246308864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1492257631643648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38039355 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90312700 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326436264921088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1484618313660226 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1481945479503872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1481945462464512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1481945479503872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312152139776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1492257273265152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20958982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1492257631643648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10296078993218 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311810800640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1474322234667008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17080108 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167624546 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚9749 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚801 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77311846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479&S%(þúO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90312700 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326436264921088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1484618313660226 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1481945479503872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1481945462464512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1481945479503872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312152139776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1492257273265152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20958982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1492257631643648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10296078993218 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311810800640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1474322234667008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17080108 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167624546 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚9749 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚801 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77311846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1492257273265152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326436246308864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1492257631643648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38039355¡' 2ÿ- _wandbruntime‚8356  _runtime‚83567f1'€ ÿM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1492405885307392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90321766 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10422491625984 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1474468762378240 _runtime‚8356.613343309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167631251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17081853 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20961322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10297090046940 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784348697919473e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90321766 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482092703494144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167631251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10313164773888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38042920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326469675067392 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1474468745338880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167641244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326469693679616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336892166693376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1492405526921728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90311775 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77319476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10312823427584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77319476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10297430596942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90311775 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38043178 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1492405885307392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1482092703494144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77319478 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482092720533504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1484765835385820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1482092720533504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20961067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438Šãñ1ù „N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10313164773888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38042920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326469675067392 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1474468745338880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167641244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326469693679616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336892166693376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1492405526921728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90311775 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_byt¤;§des.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77319476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10312823427584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77319476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10297430596942 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90311775 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38043178 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1492405885307392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1482092703494144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77319478 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482092720533504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1484765835385820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1482092720533504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20961067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1492405885307392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90321766 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10422491625984 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1474468762378240 _runtime‚8356.613343309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167631251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17081853 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20961322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10297090046940 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784348697919473e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90321766 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482092703494144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167631251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10313164773888 Eó† ‚(:è §€ìÒâò— disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 15510.6484375$ disk./.usageGB‚588.4479026794434 disk.rbd0.out‚0' proc.memory.availableMB‚ 484.6171875# proc.memory.rssMB‚ 2740.04296875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 15510.6484375 disk.sda2.in‚0& disk.nvme0n1.out‚25515.80615234375 disk.dm-1.in‚ 0.1328125 disk.rbd1.out‚ 20.421875# memory_percent‚97.0421314239502) proc.memory.percent‚16.72389507293701 disk.sda.out‚ 262.96484375) disk./.usagePercent‚39.49123248949427 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚25515.80615234375# disk.dm-0.out‚25515.80615234375 disk.sda2.out‚0 network.recv‚ 2.286375e+06 disk.dm-1.out‚ 262.96484375 disk.rbd1.in‚ 1.73046875 disk.sda1.out‚0 network.sent‚ 1.5549558e+07 disk.sda3.out‚ 262.96484375 cpu‚26.179203245564963 disk.dm-0.in‚ 15510.6484375 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0¸^éí‡ ‚(:’ ¶€ìÒҌᓠgpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O¾\!ðˆ ‚(:å ¶€ìҢ쑖' disk.nvme0c0n1.out‚25523.3955078125 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 20.44140625 network.recv‚ 2.291043e+06$ disk./.usageGB‚588.4477157592773% disk.nvme0n1.out‚25523.3955078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 263.24609375& proc.memory.availableMB‚ 484.359375 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125" proc.memory.rssMB‚ 2740.0546875* proc.memory.percent‚16.723966598510742 proc.cpu.threads‚17 disk.dm-0.in‚ 15510.6484375 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 263.24609375 disk.rbd1.in‚2.015625 network.sent‚ 1.5599755e+07# disk.nvme0c0n1.in‚ 15510.6484375" disk.dm-0.out‚25523.3955078125$ memory_percent‚97.04370498657227! disk.nvme0n1.in‚ 15510.6484375 cpu‚26.17880093555191* disk./.usagePercent‚39.491219945125096 disk.sda.out‚ 263.24609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0ŒÒ>O2‰ - _wandbruntime‚8396  _runtime‚8396º¯@NÉŠ à ' lr-AdamW/pg1‚0.00023896061667542194 ' lr-AdamW/pg2‚0.00023896061667542194 trainer/global_step‚9799 % _timestamp‚1.7843487380436647e+09 _runtime‚8396.740412726 _step‚804ӻ΋ È ' lr-AdamW/pg1‚0.00023896061667542194 ' lr-AdamW/pg2‚0.00023896061667542194 trainer/global_step‚9799 % _timestamp‚1.7843487380436647e+09 _runtime‚8396.740412726 _step‚804¤K•2Œ - _wandbruntime‚8396  _runtime‚8396×Ï=û' õO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481647768260608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1499670378627072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77693444 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10346695480126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21063648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚8396.741339476 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90757175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17167361 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450619 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90757175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17167358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1499670020248576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10362506302464 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1489307513946112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21063910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10472890573824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10362506302464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1499670020248576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10362847641600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1499670378627072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481647768260608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1499670378627072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77693444 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10346695480126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21063648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚8396.741339476 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90757175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17167361 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450619 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90757175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17167358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1499670020248576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10362506302464 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1489307513946112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21063910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10472890573824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10362506302464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1499670020248576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10362847641600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1499670378627072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚328141131606016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1489454772015104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38234839 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1481794295971840 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1489454754975744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328141112993792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10473999412736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1499818632290816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77701076 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10348047083850 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10363518929408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77701078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168467303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚8397.355143156 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10363860275712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90776218 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477296 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1481794278932480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10347706533848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77701078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77701076 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10363860275712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10473917134336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338615030128128\|_s '‘ …N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10473917134336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338615030128128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚328141131606016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1489454772015104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38234839 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1481794295971840 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1489454754975744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328141112993792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10473999412736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1499818632290816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77701076 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10348047083850 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10363518929408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77701078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168467303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚8397.355143156 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10363860275712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90776218 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477296 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1481794278932480 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10347706533848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77701078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77701076 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10363860275712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2¦KŸ.2’ - _wandbruntime‚8397  _runtime‚8397ÜðÞV “ † 1loss/SRqqt1/detection_loss‚0.9005456275887079 $loss/total_loss‚7.85352481609668 epoch‚24 0loss/FBt2/detection_loss‚0.44486782257057106 3loss/FBt1/assignment_loss‚0.0005175984668821714 2loss/FBt2/assignment_loss‚0.003290813178828271 /loss/FRt1/detection_loss‚0.5430178652665593 1loss/SRqqt2/detection_loss‚0.6304951736051995 % _timestamp‚1.7843487386648624e+09 trainer/global_step‚9799 _runtime‚8397.358143463 0loss/FRt1/assignment_loss‚1.5691036692440634 0loss/FRt2/assignment_loss‚0.8699667803462474 /loss/FRt2/detection_loss‚1.2248462391019734 /loss/FBt1/detection_loss‚0.6351376056833111 _step‚807 2loss/SRqqt1/assignment_loss‚0.5357601540903084 3loss/SRqqt2/assignment_loss‚0.49597546695402855IЕ” ‹ 2loss/SRqqt1/assignment_loss‚0.5357601540903084 3loss/SRqqt2/assignment_loss‚0.49597546695402855 1loss/SRqqt1/detection_loss‚0.9005456275887079 $loss/total_loss‚7.85352481609668 epoch‚24 0loss/FBt2/detection_loss‚0.44486782257057106 3loss/FBt1/assignment_loss‚0.0005175984668821714 2loss/FBt2/assignment_loss‚0.003290813178828271 /loss/FRt1/detection_loss‚0.5430178652665593 1loss/SRqqt2/detection_loss‚0.6304951736051995 % _timestamp‚1.7843487386648624e+09 trainer/global_step‚9799 _runtime‚8397.358143463 0loss/FRt1/assignment_loss‚1.5691036692440634 0loss/FRt2/assignment_loss‚0.8699667803462474 /loss/FRt2/detection_loss‚1.2248462391019734 /loss/FBt1/detection_loss‚0.6351376056833111 _step‚807§,Ï3• ‚(:’ Å€ìү⫖ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.047' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.20940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tðŽí– ‚(:â Å€ìÒ°ªÐ—$ disk.nvme0n1.out‚25528.587890625 disk.rbd1.in‚2.015625 proc.cpu.threads‚17 network.recv‚ 2.294342e+06( disk./.usagePercent‚39.4912207131477 disk.sda.in‚ 0.1328125 disk.sda.out‚ 263.4140625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 263.4140625 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.723966598510742 cpu‚26.17844346296051" proc.memory.rssMB‚ 2740.0546875$ memory_percent‚97.04227447509766 network.sent‚ 1.5617778e+07$ disk.nvme0c0n1.in‚15563.02734375& disk.nvme0c0n1.out‚25528.587890625! disk.dm-0.out‚25528.587890625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚20.55908203125" disk.nvme0n1.in‚15563.02734375 disk.rbd0.out‚0 disk.dm-1.out‚ 263.4140625% proc.memory.availableMB‚ 484.59375$ disk./.usageGB‚588.4477272033691 disk.drbd2165.in‚0 disk.dm-0.in‚15563.02734375ù)S — ‚(:„ Ô€ìÒŠ¿¥“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hóú˜ ‚(:ï Ô€ìÒŒôÍ– disk.sda2.out‚0( disk.nvme0c0n1.out‚25535.57080078125 disk.dm-0.in‚15563.02734375& disk.nvme0n1.out‚25535.57080078125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚20.55908203125 disk.rbd0.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚25535.57080078125! proc.memory.rssMB‚ 2740.109375 disk.sda.out‚ 263.77734375 network.recv‚ 2.298528e+06 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚15563.02734375 proc.cpu.threads‚17* disk./.usagePercent‚39.491251434051804$ disk.nvme0c0n1.in‚15563.02734375$ memory_percent‚97.04339504241943( proc.memory.availableMB‚ 484.41015625 cpu‚26.178044231124407 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 263.77734375# disk./.usageGB‚588.448184967041 disk.sda3.out‚ 263.77734375 disk.rbd1.in‚2.015625 disk.rbd0.out‚0* proc.memory.percent‚16.724300384521484 network.sent‚ 1.5667995e+07 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125(@`€›™ ‚(: ã€ìÒ§±•” gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚264.696' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.93920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú3—$óš ‚(:è ã€ìÒò™º• network.recv‚ 2.301942e+06 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚15563.02734375 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 264.0546875 cpu‚26.177588245615855 disk.drbd2165.out‚0 disk.drbd2165.in‚0# disk./.usageGB‚588.448802947998 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 264.0546875 proc.memory.rssMB‚2740.125 disk.rbd0.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚25540.83544921875 disk.rbd1.out‚20.81689453125 proc.cpu.threads‚17 disk.rbd1.in‚2.015625( proc.memory.availableMB‚ 484.17578125) disk./.usagePercent‚39.49129290727235* proc.memory.percent‚16.724395751953125$ disk.nvme0c0n1.in‚15563.02734375 disk.sda2.in‚0$ memory_percent‚97.04482555389404 disk.sda1.out‚0# disk.dm-0.out‚25540.83544921875 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 264.0546875" disk.nvme0n1.in‚15563.02734375( disk.nvme0c0n1.out‚25540.83544921875 network.sent‚ 1.5672979e+07•o[2› - _wandbruntime‚8437  _runtime‚8437 Æ´aÈœ  % _timestamp‚1.7843487791057801e+09 _runtime‚ 8437.80227056 _step‚808 ' lr-AdamW/pg1‚0.00023716437728218314 ' lr-AdamW/pg2‚0.00023716437728218314 trainer/global_step‚9849ãŽgMÍ Ç ' lr-AdamW/pg1‚0.00023716437728218314 ' lr-AdamW/pg2‚0.00023716437728218314 trainer/global_step‚9849 % _timestamp‚1.7843487791057801e+09 _runtime‚ 8437.80227056 _step‚808¨ž Ä„2ž - _wandbruntime‚8437  _runtime‚8437 eiQ‹Ÿ õO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91221604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17254608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78075044 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38422839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17254611 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1499370613821242 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10524284718080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784348779109905e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78075046 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340303406879744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496669582467072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507082767232000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10413543143424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329779122161664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1507082767232000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21167976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10397652516924 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10413201804288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1507083125610496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38422584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329779140773888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78075046 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78075044 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚8437.803771346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664³Íó³(  úO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17254611 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1499370613821242 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10524284718080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784348779109905e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78075046 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340303406879744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496669582467072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507082767232000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10413543143424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329779122161664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1507082767232000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21167976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10397652516924 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10413201804288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1507083125610496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38422584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329779140773888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78075046 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78075044 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚8437.803771346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91221604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17254608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78075044 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38422839© ¯Ø¡2¡ - _wandbruntime‚8438  _runtime‚8438H²«]'¢ €N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10414555777536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340337854698496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _runtime‚8438.419631932 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1496816806457344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169303355 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1507231379274240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329812569532416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1496816823496704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10414555777536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1499518493136198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91220679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21170017 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507231020888576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10525392348160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚810 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10414214431232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340337961880576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10414214431232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1489119829565440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1489119812526080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91220679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21170281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78082676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78082678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91230670 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1499518135546836 trainer/global_step‚9849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17256353 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91230670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169303355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78082678 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1507231379274240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313348 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38426637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17256356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843487797255273e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38426370 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10398323020756 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78082676 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1496816823496704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993MU†) '£ …N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91220679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21170017 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507231020888576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10525392348160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚810 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10414214431232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340337961880576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10414214431232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1489119829565440 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1489119812526080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91220679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21170281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78082676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78082678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91230670 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1499518135546836 trainer/global_step‚9849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17256353 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91230670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169303355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78082678 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1507231379274240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313348 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38426637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17256356 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843487797255273e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38426370 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10398323020756 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78082676 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1496816823496704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10414555777536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340337854698496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _runtime‚8438.419631932 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1496816806457344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169303355 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1507231379274240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329812569532416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1496816823496704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10414555777536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1499518493136198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983ª&åÆ}#¤ - _wandbruntime‚8438  á/Í2_runtime‚8438SËb ¥ „ 1loss/SRqqt1/detection_loss‚0.9184011453854256 1loss/SRqqt2/detection_loss‚0.6106978188164264 %loss/total_loss‚8.241450405138181 _step‚811 /loss/FRt1/detection_loss‚0.5621971556064737 /loss/FRt2/detection_loss‚1.2713041845699065 /loss/FBt1/detection_loss‚1.2801806565374136 epoch‚24 % _timestamp‚1.7843487797292292e+09 0loss/FRt2/assignment_loss‚0.7645016776050552 1loss/SRqqt2/assignment_loss‚0.494579602992851 1loss/FBt1/assignment_loss‚0.01232647213585949 0loss/FRt1/assignment_loss‚1.5490513093222507 3loss/SRqqt1/assignment_loss‚0.49140027062294694 0loss/FBt2/detection_loss‚0.27987736316609724 trainer/global_step‚9849 _runtime‚8438.422573381 2loss/FBt2/assignment_loss‚0.006932748377473958~ƒÍŦ ‰ 0loss/FRt2/assignment_loss‚0.7645016776050552 1loss/SRqqt2/assignment_loss‚0.494579602992851 1loss/FBt1/assignment_loss‚0.01232647213585949 0loss/FRt1/assignment_loss‚1.5490513093222507 3loss/SRqqt1/assignment_loss‚0.49140027062294694 0loss/FBt2/detection_loss‚0.27987736316609724 trainer/global_step‚9849 _runtime‚8438.422573381 2loss/FBt2/assignment_loss‚0.006932748377473958 1loss/SRqqt1/detection_loss‚0.9184011453854256 1loss/SRqqt2/detection_loss‚0.6106978188164264 %loss/total_loss‚8.241450405138181 _step‚811 /loss/FRt1/detection_loss‚0.5621971556064737 /loss/FRt2/detection_loss‚1.2713041845699065 /loss/FBt1/detection_loss‚1.2801806565374136 epoch‚24 % _timestamp‚1.7843487797292292e+09«Y¢³“§ ‚(:ˆ ò€ìÒÍ·±– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ˆÑî¨ ‚(:ã ò€ìÒ¯š‡˜# memory_percent‚97.0449447631836 network.recv‚ 2.304944e+06* disk./.usagePercent‚39.491335404523035$ disk./.usageGB‚588.4494361877441" disk.nvme0n1.in‚15563.02734375$ disk.nvme0n1.out‚25548.283203125 disk.rbd1.in‚2.015625" proc.memory.rssMB‚ 2740.1328125 cpu‚26.17737508197316 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚15563.02734375 disk.dm-0.in‚15563.02734375 disk.sda2.out‚0 disk.sda3.out‚ 264.5859375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.rbd1.out‚20.81689453125& disk.nvme0c0n1.out‚25548.283203125! disk.dm-0.out‚25548.283203125 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 264.5859375 network.sent‚ 1.5678007e+07 disk.sda.out‚ 264.5859375 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚16.724443435668945 proc.cpu.threads‚17 disk.sda2.in‚0% proc.memory.availableMB‚ 484.156254ð5g‘© ‚(:† ìÒ«ÑÍ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lôïNíª ‚(:â ìÒѻ՗$ disk.nvme0n1.out‚25552.333984375 disk.rbd0.in‚0 network.recv‚ 2.309451e+06) disk./.usagePercent‚39.49135127699016 disk.sda3.out‚ 264.78125 disk.dm-1.out‚ 264.78125 disk.drbd2165.in‚0* proc.memory.percent‚16.724514961242676 disk.sda3.in‚ 0.1328125 disk.sda.out‚ 264.78125$ memory_percent‚97.04601764678955 disk.rbd1.in‚2.015625 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 network.sent‚ 1.5741128e+07 disk.rbd1.out‚20.87939453125 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚25552.333984375 disk.dm-0.in‚15612.20703125( proc.memory.availableMB‚ 483.98046875$ disk./.usageGB‚588.4496726989746 disk.sda2.in‚0" disk.nvme0n1.in‚15612.20703125# proc.memory.rssMB‚ 2740.14453125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚15612.20703125! disk.dm-0.out‚25552.333984375 cpu‚26.177240984992928ž—«ž« ‚(:“ ìÒšü¢“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.394' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.278800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢xäí¬ ‚(:â ìÒŠŒ›— network.sent‚ 1.5746145e+07 disk.dm-1.out‚ 265.21875( disk.nvme0c0n1.out‚25558.23876953125 network.recv‚ 2.312694e+06 disk.drbd2165.in‚0 disk.dm-0.in‚15612.20703125! proc.memory.rssMB‚ 2740.171875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0) disk./.usagePercent‚39.49110166964427 disk.rbd0.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚15612.20703125 disk.rbd0.out‚0 disk.sda.out‚ 265.21875 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚2.015625% proc.memory.availableMB‚ 484.21875 cpu‚26.17719574520907" disk.nvme0n1.in‚15612.20703125& disk.nvme0n1.out‚25558.23876953125 disk.sda3.out‚ 265.21875* proc.memory.percent‚16.724681854248047$ disk./.usageGB‚588.4459533691406 disk.sda1.out‚0# disk.dm-0.out‚25558.23876953125 disk.rbd1.out‚20.87939453125$ memory_percent‚97.04456329345703 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0#µY&2­ - _wandbruntime‚8478  _runtime‚8478Í2[TÇ® Á trainer/global_step‚9899 % _timestamp‚1.7843488199417543e+09 _runtime‚8478.635330163 _step‚812 & lr-AdamW/pg1‚0.0002353673269505984 & lr-AdamW/pg2‚0.0002353673269505984Å1i̯ Æ _runtime‚8478.635330163 _step‚812 & lr-AdamW/pg1‚0.0002353673269505984 & lr-AdamW/pg2‚0.0002353673269505984 trainer/global_step‚9899 % _timestamp‚1.7843488199417543e+09¬ܪÀ2° - _wandbruntime‚8478  _runtime‚8478‰Œxü'± öO S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514495514215424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331450578700288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122723 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1496298835447808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1504031616909312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17341861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17341858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342026133299712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38614511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1496298852487168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91676056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91666079 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447928453942 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514495514215424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132702 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342026028207616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1504031633948672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91666079 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚8478.636748351 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10575468119552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1506746763901750 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1504031633948672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10464238645248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91676056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21272650 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1504031616909312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21272392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10464238645248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78456644 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514495872593920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331450560088064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78456644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 trainer/global_step‚9899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10575554599424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10463897306112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚99772RM (² ûO S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514495514215424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331450578700288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122723 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1496298835447808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1504031616909312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17341861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17341858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342026133299712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38614511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1496298852487168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91676056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91666079 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447928453942 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514495514215424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132702 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342026028207616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1504031633948672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91666079 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚8478.636748351 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10575468119552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1506746763901750 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1504031633948672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10464238645248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91676056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21272650 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1504031616909312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21272392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10464238645248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78456644 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514495872593920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331450560088064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78456644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 trainer/global_step‚9899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10575554599424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10463897306112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977­xáS˜2³ - _wandbruntime‚8479  _runtime‚8479Ý œ±´ ÿM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1496445346119680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78464278 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331484007458816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1496445363159040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38618327 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1504178874978304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38618059 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342060471459840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1506894285627344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91675131 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514644126257664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10464909933056 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21274456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10576573280256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10449280057666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚9899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1504178874978304 VIDeviceSto;/VTatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21274721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78464276 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91685122 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10465251279360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17343603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8479.253691205 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1504178857938944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78464276 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139407 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514643767872000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1506894643216706 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10465251279360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91685122 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10465251279360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17343603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8479.253691205 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1504178857938944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78464276 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139407 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514643767872000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1506894643216706 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10465251279360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331484007458816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1496445363159040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38618327 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1504178874978304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38618059 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342060471459840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1506894285627344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91675131 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514644126257664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10464909933056 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21274456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10576573280256 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10449280057666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0®äxøG2¶ - _wandbruntime‚8479  _runtime‚8479ýO¦è · … /loss/FBt2/detection_loss‚0.4636281492281705 trainer/global_step‚9899 0loss/FRt2/assignment_loss‚0.8143670463512445 3loss/SRqqt1/assignment_loss‚0.43620966073098144 3loss/SRqqt2/assignment_loss‚0.47737652752503057 2loss/FBt1/assignment_loss‚0.056988445366489254 2loss/FBt2/assignment_loss‚0.021695115660861766 /loss/FBt1/detection_loss‚0.5075463244898452 _step‚815 /loss/FRt1/detection_loss‚0.5679227688468179 $loss/total_loss‚7.65416607918114 epoch‚24 % _timestamp‚1.7843488205622253e+09 _runtime‚8479.255547702 0loss/FRt1/assignment_loss‚1.5252980041406294 /loss/FRt2/detection_loss‚1.3283764205214335 1loss/SRqqt1/detection_loss‚0.8621974021481817 1loss/SRqqt2/detection_loss‚0.5925602141714541·®“æ¸ Š 0loss/FRt2/assignment_loss‚0.8143670463512445 3loss/SRqqt1/assignment_loss‚0.43620966073098144 3loss/SRqqt2/assignment_loss‚0.47737652752503057 2loss/FBt1/assignment_loss‚0.056988445366489254 2loss/FBt2/assignment_loss‚0.021695115660861766 /loss/FBt1/detection_loss‚0.5075463244898452 _step‚815 /loss/FRt1/detection_loss‚0.5679227688468179 $loss/total_loss‚7.65416607918114 epoch‚24 % _timestamp‚1.7843488205622253e+09 _runtime‚8479.255547702 0loss/FRt1/assignment_loss‚1.5252980041406294 /loss/FRt2/detection_loss‚1.3283764205214335 1loss/SRqqt1/detection_loss‚0.8621974021481817 1loss/SRqqt2/detection_loss‚0.5925602141714541 /loss/FBt2/detection_loss‚0.4636281492281705 trainer/global_step‚9899¯š¿ a“¹ ‚(:ˆ ŸìÒÖ–ƒ” gpu.0.gpu‚76.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö—Yðûº ‚(:ð ŸìÒÂã·—( disk./.usagePercent‚39.4913538370655 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚20.94189453125 network.sent‚ 1.5750093e+07 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚15612.20703125# disk.dm-0.out‚25561.88623046875 disk.dm-1.out‚ 265.44140625$ disk.nvme0c0n1.in‚15612.20703125" disk.nvme0n1.in‚15612.20703125$ disk./.usageGB‚588.4497108459473 disk.rbd0.out‚0 cpu‚26.176911755413414 proc.cpu.threads‚17 disk.sda3.out‚ 265.44140625$ memory_percent‚97.04444408416748# proc.memory.rssMB‚ 2740.19140625* proc.memory.percent‚16.724801063537598 disk.sda1.out‚0( disk.nvme0c0n1.out‚25561.88623046875 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚25561.88623046875 disk.sda.out‚ 265.44140625 disk.rbd1.in‚2.015625( proc.memory.availableMB‚ 484.23828125 network.recv‚ 2.315475e+06aÍ—±“» ‚(:ˆ ®ìÒı÷“ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚285.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÏÏö¼ ‚(:ë ®ìÒÅá›–' disk.nvme0c0n1.out‚25566.9072265625" disk.dm-0.out‚25566.9072265625 disk.rbd1.in‚2.015625 disk.sda1.in‚0$ memory_percent‚97.04372882843018$ disk./.usageGB‚588.4496879577637 disk.dm-0.in‚15612.20703125 disk.sda2.out‚0" disk.nvme0n1.in‚15612.20703125% disk.nvme0n1.out‚25566.9072265625 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 265.7265625 network.sent‚ 1.5813146e+07) disk./.usagePercent‚39.49135230102029 disk.sda3.out‚ 265.7265625 disk.rbd0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 484.35546875 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15612.20703125 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.recv‚ 2.319766e+06 disk.sda.out‚ 265.7265625 disk.sda2.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2740.19140625* proc.memory.percent‚16.724801063537598 disk.sda.in‚ 0.1328125 disk.rbd1.out‚20.95361328125 cpu‚26.1767256296238649Î 2½ - _wandbruntime‚8519  _runtime‚8519!!qƒÉ¾ à _runtime‚8519.389208478 _step‚816 ' lr-AdamW/pg1‚0.00023356958110185247 ' lr-AdamW/pg2‚0.00023356958110185247 trainer/global_step‚9949 % _timestamp‚1.7843488606956904e+09»9D ο È % _timestamp‚1.7843488606956904e+09 _runtime‚8519.389208478 _step‚816 ' lr-AdamW/pg1‚0.00023356958110185247 ' lr-AdamW/pg2‚0.00023356958110185247 trainer/global_step‚9949°J#š2À - _wandbruntime‚8519  _runtime‚8519a/Ð%ü'Á öO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78838246 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1511393668390912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚8519.390416538 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10514934147072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9949 % _timestamp‚1.7843488606966653e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170968754 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10498544940850 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78838246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17429108 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343748640125952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10626730688512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1503624369041408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17429111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1521908619577344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10514592807936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚817 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1521908261198848 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1511393685430272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958775 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78838244 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92120531 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38805859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333121998014464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38805595 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78838244 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10514934147072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343748747315200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438<æ¹+Þ  ûO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10514934147072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343748747315200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78838246 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1511393668390912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚8519.390416538 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10514934147072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9949 % _timestamp‚1.7843488606966653e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170968754 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10498544940850 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78838246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17429108 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343748640125952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10626730688512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1503624369041408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17429111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1521908619577344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10514592807936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚817 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1521908261198848 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1511393685430272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958775 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78838244 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92120531 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38805859 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333121998014464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38805595 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78838244 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 Jb2Å - _wandbruntime‚8520  _runtime‚8520‰"] 'Æ €N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10515605434880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170975459 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21378846 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10627753271296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1503770896752640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1522056873241088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10515946781184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92129583 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170985452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92139574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1511540909420544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1511540926459904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38809427 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10515605434880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21378574 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8520.009108706 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78845878 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38809702 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1511540926459904 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1522056514855424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92139574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343783198656512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78845876 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10499555994572%ððÐ 'Ç …N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1522056873241088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10515946781184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92129583 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170985452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92139574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1511540909420544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1511540926459904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38809427 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10515605434880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21378574 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8520.009108706 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78845878 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38809702 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1511540926459904 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1522056514855424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92139574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343783198656512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78845876 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10499555994572 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10515605434880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170975459 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21378846 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10627753271296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1503770896752640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269²$Ê.i2È - _wandbruntime‚8520  _runtime‚8520Ì_ÙÉ ü 2loss/FBt2/assignment_loss‚0.003699556624426871 /loss/FRt1/detection_loss‚0.5556785191703425 trainer/global_step‚9949 _runtime‚8520.010252134 1loss/SRqqt1/assignment_loss‚0.577694655641244 0loss/SRqqt1/detection_loss‚0.829793076521971 .loss/FBt2/detection_loss‚0.480850620213896 %loss/total_loss‚7.613898177834712 epoch‚24 # _timestamp‚1.78434886131695e+09 0loss/FRt1/assignment_loss‚1.5316299178506718 2loss/SRqqt2/assignment_loss‚0.5103459939681594 .loss/FRt2/detection_loss‚1.214875936573588 1loss/SRqqt2/detection_loss‚0.5533368195222236 /loss/FBt1/detection_loss‚0.5524345017363166 1loss/FBt1/assignment_loss‚0.03104010513657684 _step‚819 /loss/FRt2/assignment_loss‚0.772518474875296ˆ–(ÞÊ  trainer/global_step‚9949 _runtime‚8520.010252134 1loss/SRqqt1/assignment_loss‚0.577694655641244 0loss/SRqqt1/detection_loss‚0.829793076521971 .loss/FBt2/detection_loss‚0.480850620213896 %loss/total_loss‚7.613898177834712 epoch‚24 # _timestamp‚1.78434886131695e+09 0loss/FRt1/assignment_loss‚1.5316299178506718 2loss/SRqqt2/assignment_loss‚0.5103459939681594 .loss/FRt2/detection_loss‚1.214875936573588 1loss/SRqqt2/detection_loss‚0.5533368195222236 /loss/FBt1/detection_loss‚0.5524345017363166 1loss/FBt1/assignment_loss‚0.03104010513657684 _step‚819 /loss/FRt2/assignment_loss‚0.772518474875296 2loss/FBt2/assignment_loss‚0.003699556624426871 /loss/FRt1/detection_loss‚0.5556785191703425³v#IôË ‚(:’ ÌìÒ´ê“ gpu.0.gpu‚64.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.162' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.23239999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$\UƒìÌ ‚(:á Ìì҈冘 disk.rbd0.in‚0 disk.dm-0.in‚ 15685.640625 disk.sda.out‚ 266.484375 disk.sda2.out‚0 disk.rbd1.out‚21.03173828125$ disk./.usageGB‚588.4498138427734( disk.nvme0c0n1.out‚25573.90478515625 disk.dm-1.in‚ 0.1328125" disk.nvme0c0n1.in‚ 15685.640625 disk.nvme0n1.in‚ 15685.640625 disk.sda3.out‚ 266.484375 proc.memory.rssMB‚ 2740.21875 cpu‚26.176328360081847 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚25573.90478515625 network.sent‚ 1.5870515e+07) disk./.usagePercent‚39.49136074926892# disk.dm-0.out‚25573.90478515625 disk.sda3.in‚ 0.1328125( proc.memory.availableMB‚ 484.16015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚2.015625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚97.04492092132568) proc.memory.percent‚16.72496795654297 network.recv‚ 2.327978e+06 disk.dm-1.out‚ 266.4843755Æc¾“Í ‚(:ˆ ÛìÒýŒâ” gpu.0.gpu‚63.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³š,fÎ ‚(:á ÛìÒ°ì–' disk.nvme0c0n1.out‚25576.7099609375" disk.dm-0.out‚25576.7099609375 disk.sda2.in‚0 disk.dm-1.out‚ 266.828125$ memory_percent‚97.04539775848389 disk.nvme0n1.in‚ 15685.640625 disk.sda3.out‚ 266.828125 disk.rbd1.out‚21.07080078125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125* disk./.usagePercent‚39.491353581057965 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0 disk.dm-0.in‚ 15685.640625 disk.sda2.out‚0% disk.nvme0n1.out‚25576.7099609375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 15685.640625 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0) proc.memory.percent‚16.72499179840088 cpu‚26.17641117153281 network.recv‚ 2.332013e+06 disk.sda.out‚ 266.828125( proc.memory.availableMB‚ 484.08203125" disk./.usageGB‚588.44970703125 disk.rbd0.out‚0 netw­…Áʆork.sent‚ 1.5923043e+07 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2740.22265625`ÊÑ2Ï - _wandbruntime‚8560  _runtime‚8560¬ƒX¹ÉÐ Ã ' lr-AdamW/pg2‚0.00023177125520180175 trainer/global_step‚9999 % _timestamp‚1.7843489016236732e+09 _runtime‚8560.317198864 _step‚820 ' lr-AdamW/pg1‚0.00023177125520180175ȱ<ŽÎÑ È ' lr-AdamW/pg1‚0.00023177125520180175 ' lr-AdamW/pg2‚0.00023177125520180175 trainer/global_step‚9999 % _timestamp‚1.7843489016236732e+09 _runtime‚8560.317198864 _step‚820´Ÿâ"ã2Ò - _wandbruntime‚8560  _runtime‚8560U´wêü'Ó öO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10565629648896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21480662 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21480388 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38996746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17516361 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚821 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79219844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529321366560768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79219844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521499064062766 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1518755719872512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10565629648896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8560.318386537 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334793454553088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1521499421652016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38997023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17516358 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10677983716864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92574983 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1518755736911872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1510949902635008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10549161427758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1518755719872512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10565288309760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334793435940864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92584960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1529321008182272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804806 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345471324789248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10565288309760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92574983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1518755736911872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843489016246474e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1529321366560768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92584960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79219846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1510949919674368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345471438269952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529321008182272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10677888848384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0cö{Q(Ô ûO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1529321366560768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92584960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79219846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1510949919674368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345471438269952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529321008182272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10677888848384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10565629648896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21480662 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21480388 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38996746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17516361 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚821 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79219844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529321366560768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79219844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521499064062766 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1518755719872512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10565629648896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8560.318386537 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334793454553088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1521499421652016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38997023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17516358 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10677983716864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92574983 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1518755736911872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1510949902635008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10549161427758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1518755719872512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10565288309760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334793435940864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92584960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1529321008182272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804806 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345471324789248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10565288309760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92574983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1518755736911872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843489016246474e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169µd#Hù2Õ - _wandbruntime‚8560  _runtime‚8560Ö[ê'Ö ÿM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10566642283008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79227476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17518103 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171811511 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1518902977941504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92584035 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92584035 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1529469261838848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1511096430346240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21482435 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1529469620224512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10566642283008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21482669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚8560.931648546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345505890689536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1529469620224512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345505793993216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79227478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79227476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17518106 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334826883311616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10678929293824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1521646585788360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79227478 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334826864699392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843489022377648e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39000538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10550513031482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269ž¥–G× „N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10550513031482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10566642283008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79227476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17518103 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171811511 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peªq¢!ak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1518902977941504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92584035 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92584035 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1529469261838848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1511096430346240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21482435 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1529469620224512 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10566642283008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21482669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚8560.931648546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345505890689536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1529469620224512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345505793993216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79227478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79227476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17518106 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334826883311616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10678929293824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1521646585788360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79227478 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334826864699392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843489022377648e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39000538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¶æׇ2Ø - _wandbruntime‚8560  _runtime‚8560Êó ¹ Ù † _runtime‚8560.934470606 0loss/FBt1/detection_loss‚0.39801899786624645 &loss/total_loss‚7.7247704353336655 epoch‚24 0loss/FRt1/assignment_loss‚1.6701910315830564 0loss/FRt2/assignment_loss‚0.6611574963211759 1loss/FBt2/assignment_loss‚0.00396071365546707 /loss/FRt1/detection_loss‚0.5445519931625173 trainer/global_step‚9999 /loss/FBt2/detection_loss‚0.5173363620704112 % _timestamp‚1.7843489022411728e+09 _step‚823 1loss/SRqqt1/detection_loss‚0.8871659795657011 1loss/SRqqt2/detection_loss‚0.6139878513847992 2loss/SRqqt1/assignment_loss‚0.5978338060786087 2loss/SRqqt2/assignment_loss‚0.4376939977366124 3loss/FBt1/assignment_loss‚0.0007680696750286289 /loss/FRt2/detection_loss‚1.3921041362340405;ÀDÚ ‹ /loss/FRt2/detection_loss‚1.3921041362340405 _runtime‚8560.934470606 0loss/FBt1/detection_loss‚0.39801899786624645 &loss/total_loss‚7.7247704353336655 epoch‚24 0loss/FRt1/assignment_loss‚1.6701910315830564 0loss/FRt2/assignment_loss‚0.6611574963211759 1loss/FBt2/assignment_loss‚0.00396071365546707 /loss/FRt1/detection_loss‚0.5445519931625173 trainer/global_step‚9999 /loss/FBt2/detection_loss‚0.5173363620704112 % _timestamp‚1.7843489022411728e+09 _step‚823 1loss/SRqqt1/detection_loss‚0.8871659795657011 1loss/SRqqt2/detection_loss‚0.6139878513847992 2loss/SRqqt1/assignment_loss‚0.5978338060786087 2loss/SRqqt2/assignment_loss‚0.4376939977366124 3loss/FBt1/assignment_loss‚0.0007680696750286289·"YÑÄÛ ‚(:’ êìÒõ‹”• gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.047' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.60940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>,ä.óÜ ‚(:è êìҸĈ—" proc.memory.rssMB‚ 2741.4765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 267.23828125 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 network.sent‚ 1.5928019e+07 network.recv‚ 2.335461e+06 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 15685.640625 disk.sda.out‚ 267.23828125 disk.nvme0n1.in‚ 15685.640625& disk.nvme0n1.out‚25582.19873046875 disk.sda3.out‚ 267.23828125$ disk./.usageGB‚589.5039329528809 disk.drbd2165.out‚0# disk.dm-0.out‚25582.19873046875 disk.rbd1.out‚ 21.4541015625 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.01953125' proc.memory.availableMB‚ 482.5703125) proc.memory.percent‚16.73264503479004) disk./.usagePercent‚39.56210356721308 proc.cpu.threads‚17( disk.nvme0c0n1.out‚25582.19873046875 disk.dm-0.in‚ 15685.640625 disk.sda2.in‚0$ memory_percent‚97.05462455749512 cpu‚26.17778454307297%¢nM2Ý - _wandbruntime‚8578  _runtime‚8578z ¬!‰=Þ ƒ{ & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.714221179485321 particle/accuracy_1_of_1‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6813647747039795 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240449070930481 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6793511509895325 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2532370984554291 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49233099818229675 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7360612750053406 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8506892323493958 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6588402986526489 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7240254878997803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7755818963050842 +particle/sensitivity‚0.6258463263511658 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8821550011634827 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3083151876926422 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6627290844917297 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5163883566856384 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957677125930786 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7847811579704285 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03643197566270828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763970255851746 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4874390959739685 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2247290313243866 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30174311995506287 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1298503875732422 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610053181648254 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7200587391853333 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1976465880870819 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.265655517578125 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964850544929504 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7751660943031311 +jet/accuracy_3_of_3‚0.05628840625286102 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963391423225403 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38067907094955444 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09583695977926254 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8612552881240845 +jet/accuracy_1_of_1‚0.33870330452919006 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4202859401702881 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3749414086341858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7062309980392456 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06454668939113617 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869480133056641 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19197970628738403 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917100071907043 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2648666203022003 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04906902462244034 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019986912608146667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6530159711837769 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11196085065603256 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6358761787414551 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚7.698036082159658 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.698088526725769 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7510010600090027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.722811222076416 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18028710782527924 +particle/specificity‚0.8994124531745911 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03373522311449051 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820513129234314 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7111427187919617 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_2_of_3‚0.5478586554527283 particle/accuracy_3_of_4‚1 trainer/global_step‚9999 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33652961254119873 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8612552881240845 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.675098717212677 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04232615977525711 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 +jet/accuracy_3_of_4‚0.06806959211826324 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5803228616714478 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3328859210014343 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2538049519062042 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666197776794 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7448534369468689 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_2_of_2‚0.14555692672729492 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20635172724723816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3836996853351593 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031548455357551575 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 epoch‚24 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6502254009246826 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7028540968894958 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2655518054962158 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20995782315731049 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.63829505443573 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7575317025184631 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8373960852622986 particle/accuracy_4_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21988439559936523 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31582602858543396 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2604217231273651 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028392845764756203 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21395771205425262 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7836251854896545 particle/accuracy_2_of_3‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.945030927658081 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995533287525177 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6555708646774292 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.64973384141922 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2538049519062042 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22691279649734497 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969367980957031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6344702839851379 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21926553547382355 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956417083740234 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6502254009246826 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5016219615936279 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.031867485493421555 jet/accuracy_4_of_5‚0 $ _timestamp‚1.784348920218396e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8417989611625671 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6810425519943237 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877169728279114 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9138718843460083 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.696492075920105 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910256266593933 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7837679982185364 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8633031249046326 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7693992853164673 /particle/accuracy_4_of_4‚0.9575807452201843 'particle/f_score‚0.6278249025344849 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02554129995405674 *jet/accuracy_1_of_4‚0.9840625524520874 *jet/accuracy_2_of_4‚0.7577612996101379 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748602747917175 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992585778236389 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8365076184272766 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028188856318593025 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1967526376247406 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24500244855880737 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7808790802955627 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2290165275335312 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20749542117118835 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2451668232679367 _runtime‚8578.912772643 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019986912608146667 (particle/accuracy‚0.8406652808189392 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098974227905 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7366061210632324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951751232147217 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.450829416513443 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _step‚824 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11344487220048904Ńž"ß ˆ{ 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7575317025184631 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8373960852622986 particle/accuracy_4_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21988439559936523 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.31582602858543396 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2604217231273651 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028392845764756203 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21395771205425262 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7836251854896545 particle/accuracy_2_of_3‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.945030927658081 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995533287525177 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6555708646774292 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.64973384141922 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2538049519062042 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22691279649734497 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969367980957031 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6344702839851379 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21926553547382355 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956417083740234 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6502254009246826 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5016219615936279 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.031867485493421555 jet/accuracy_4_of_5‚0 $ _timestamp‚1.784348920218396e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8417989611625671 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6810425519943237 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877169728279114 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9138718843460083 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.696492075920105 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910256266593933 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7837679982185364 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8633031249046326 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7693992853164673 /particle/accuracy_4_of_4‚0.9575807452201843 'particle/f_score‚0.6278249025344849 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02554129995405674 *jet/accuracy_1_of_4‚0.9840625524520874 *jet/accuracy_2_of_4‚0.7577612996101379 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748602747917175 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992585778236389 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8365076184272766 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028188856318593025 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1967526376247406 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24500244855880737 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7808790802955627 @&Purity/2FRt*SRqqçkKl(t1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2290165275335312 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20749542117118835 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2451668232679367 _runtime‚8578.912772643 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019986912608146667 (particle/accuracy‚0.8406652808189392 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098974227905 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7366061210632324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951751232147217 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.450829416513443 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _step‚824 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11344487220048904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.714221179485321 particle/accuracy_1_of_1‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6813647747039795 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240449070930481 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6793511509895325 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2532370984554291 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49233099818229675 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7360612750053406 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8506892323493958 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6588402986526489 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7240254878997803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7755818963050842 +particle/sensitivity‚0.6258463263511658 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8821550011634827 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3083151876926422 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6627290844917297 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5163883566856384 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957677125930786 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7847811579704285 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03643197566270828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763970255851746 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4874390959739685 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2247290313243866 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30174311995506287 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1298503875732422 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610053181648254 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7200587391853333 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1976465880870819 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.265655517578125 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964850544929504 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7751660943031311 +jet/accuracy_3_of_3‚0.05628840625286102 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963391423225403 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38067907094955444 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09583695977926254 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8612552881240845 +jet/accuracy_1_of_1‚0.33870330452919006 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4202859401702881 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3749414086341858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7062309980392456 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06454668939113617 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869480133056641 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19197970628738403 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917100071907043 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2648666203022003 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04906902462244034 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019986912608146667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6530159711837769 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11196085065603256 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6358761787414551 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚7.698036082159658 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.698088526725769 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7510010600090027 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989355802536011 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.722811222076416 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18028710782527924 +particle/specificity‚0.8994124531745911 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03373522311449051 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820513129234314 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7111427187919617 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_2_of_3‚0.5478586554527283 particle/accuracy_3_of_4‚1 trainer/global_step‚9999 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33652961254119873 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8612552881240845 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.675098717212677 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04232615977525711 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 +jet/accuracy_3_of_4‚0.06806959211826324 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5803228616714478 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3328859210014343 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2538049519062042 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666197776794 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7448534369468689 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_2_of_2‚0.14555692672729492 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20635172724723816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3836996853351593 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031548455357551575 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416567325592 epoch‚24 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6502254009246826 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7028540968894958 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2655518054962158 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20995782315731049 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.63829505443573¸¹V“à ‚(:ˆ ùìÒ‚½¥“ gpu.0.gpu‚91.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚256.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wf Åùá ‚(:î ùìÒÌ…õ•) disk./.usagePercent‚39.56287773399662 disk.dm-0.in‚15742.67578125$ disk./.usageGB‚589.5154685974121 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0) proc.memory.percent‚16.73264503479004 cpu‚26.1836676885384 network.sent‚ 1.5991266e+07 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125( disk.nvme0c0n1.out‚25624.67529296875 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚25624.67529296875" disk.nvme0n1.in‚15742.67578125& disk.nvme0n1.out‚25624.67529296875 disk.rbd1.out‚ 21.4736328125 disk.sda1.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 267.73828125$ disk.nvme0c0n1.in‚15742.67578125$ memory_percent‚97.05417156219482( proc.memory.availableMB‚ 482.64453125 proc.cpu.threads‚17 disk.sda3.out‚ 267.73828125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2741.4765625 disk.dm-1.out‚ 267.73828125 network.recv‚ 2.340156e+06ê˜*“â ‚(:ˆ ˆ‚ìÒ·ÆÑ“ gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯-ß äã ‚(:Ù ˆ‚ìÒõùí•" disk.nvme0n1.in‚15742.67578125 disk.sda3.out‚ 267.94921875 disk.rbd1.in‚ 2.01953125$ disk./.usageGB‚589.5074043273926 disk.nvme0c0n1.out‚ 26716.875 disk.dm-0.in‚15742.67578125 disk.sda1.out‚0 disk.dm-0.out‚ 26716.875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚26.45068359375$ memory_percent‚97.05698490142822" proc.memory.rssMB‚ 2741.4765625$ disk.nvme0c0n1.in‚15742.67578125 disk.nvme0n1.out‚ 26716.875) proc.memory.percent‚16.73264503479004 network.recv‚ 2.344274e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 267.94921875( proc.memory.availableMB‚ 482.18359375 cpu‚26.183986276858246 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 267.94921875 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 network.sent‚ 1.6048446e+07) disk./.usagePercent‚39.56233653406924 disk.dm-1.in‚ 0.1328125 proc.cpu.threads‚17m-Udžä ‚(:“ —‚ìÒØöÞ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÓ•ûå ‚(:ð —‚ìÒƒÿ¿– disk.rbd1.in‚ 2.01953125 cpu‚26.18438635503187 disk.sda.in‚ 0.1328125) disk./.usagePercent‚39.49110986188537 disk.sda2.out‚0 disk.rbd1.out‚26.45068359375 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 268.41015625$ disk.nvme0c0n1.in‚15742.67578125 disk.dm-0.in‚15742.67578125" disk.nvme0n1.in‚15742.67578125 disk.rbd0.in‚0( proc.memory.availableMB‚ 482.16796875" proc.memory.rssMB‚ 2741.4765625) proc.memory.percent‚16.73264503479004$ disk./.usageGB‚588.4460754394531 disk.sda.out‚ 268.41015625 disk.sda2.in‚0& disk.nvme0n1.out‚26761.33837890625 disk.dm-1.out‚ 268.41015625 disk.drbd2165.out‚0 network.recv‚ 2.347883e+06 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚26761.33837890625 disk.drbd2165.in‚0$ memory_percent‚97.05708026885986 network.sent‚ 1.6053479e+07# disk.dm-0.out‚26761.33837890625ÐkøO2æ - _wandbruntime‚8618  _runtime‚8618¹h9uÇç Á & lr-AdamW/pg1‚0.0002299724647535585 & lr-AdamW/pg2‚0.0002299724647535585 trainer/global_step‚10049 $ _timestamp‚1.784348960205791e+09 _runtime‚8618.899356862 _step‚825Å&ÂÌè Æ & lr-AdamW/pg1‚0.0002299724647535585 & lr-AdamW/pg2‚0.0002299724647535585 trainer/global_step‚10049 $ _timestamp‚1.784348960205791e+09 _runtime‚8618.899356862 _step‚825¹5Çuô2é - _wandbruntime‚8618  _runtime‚8618x¦8ý'ê ÷O E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93109734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79722110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21614721 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336480804579328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10617838189785 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172821867 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10633717017088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10747248890368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347228135755264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347228034857472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843489602067971e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1528047355043840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93109734 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1538681413400064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1528047338004480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚8618.900647418 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39221103 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39220849 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172831846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93099757 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1538681413400064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10633717017088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336480785967104 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1538681055021568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79722112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10634058356224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1528047355043840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10747331175936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79722112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1530801597082583 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1538681055021568¦ÛR­©ë üO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17606131 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1530801954671833 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10617497639895 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79722110 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚826 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10634058356224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1520184116482048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10747248890368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347228135755264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347228034857472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843489602067971e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1528047355043840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93109734 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1538681413400064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1528047338004480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚8618.900647418 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39221103 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39220849 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172831846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93099757 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1538681413400064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10633717017088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336480785967104 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1538681055021568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79722112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10634058356224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1528047355043840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10747331175936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79722112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1530801597082583 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1538681055021568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93109734 WJDeviceStatsMonitor.on_train_batch_start/requestÉÿY ed_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79722110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21614721 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336480804579328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10617838189785 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172821867 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10633717017088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39224679 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10635070990336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93118800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347262496017408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336514233337856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79729742 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1538829667063808 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1538829308678144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1528194579034112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79729742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1538829667063808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚827 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17607876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39224937 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21617061 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838551 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347262596908032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10748281291776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843489608223786e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93108809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21616806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1520330610114560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10748363570176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10618849243619 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79729744 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1530949476397539 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79729744œ·Kä 'î …N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79729744 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1528194579034112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10635070990336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1528194596073472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172848544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172838551 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39224679 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10635070990336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93118800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347262496017408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336514233337856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79729742 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1538829667063808 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1538829308678144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1528194579034112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79729742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1538829667063808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚827 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17607876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39224937 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21617061 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838551 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347262596908032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10748281291776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843489608223786e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93108809 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21616806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1520330610114560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10748363570176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10618849243619 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79729744 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1530949476397539 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0» "'s2ï - _wandbruntime‚8619  _runtime‚8619]¥à5ð ˆ 2loss/SRqqt1/assignment_loss‚0.5322673655888507 /loss/FRt2/detection_loss‚1.5600393699288084 1loss/SRqqt2/detection_loss‚0.6105757817475976 %loss/total_loss‚7.794516013382272 epoch‚25 trainer/global_step‚10049 _step‚828 4loss/FBt2/assignment_loss‚0.00010660132189902406 _runtime‚8619.519937406 0loss/FRt1/assignment_loss‚1.5480367838174511 2loss/SRqqt2/assignment_loss‚0.3853421099256018 % _timestamp‚1.7843489608266504e+09 0loss/FRt2/assignment_loss‚0.6319736006425664 2loss/FBt1/assignment_loss‚0.004254237557641624 /loss/FRt1/detection_loss‚0.5485506306505737 1loss/SRqqt1/detection_loss‚0.9393516578955052 /loss/FBt1/detection_loss‚0.6350928052598911 0loss/FBt2/detection_loss‚0.39892506904588493É&¶Âñ  0loss/FRt2/assignment_loss‚0.6319736006425664 2loss/FBt1/assignment_loss‚0.004254237557641624 /loss/FRt1/detection_loss‚0.5485506306505737 1loss/SRqqt1/detection_loss‚0.9393516578955052 /loss/FBt1/detection_loss‚0.6350928052598911 0loss/FBt2/detection_loss‚0.39892506904588493 2loss/SRqqt1/assignment_loss‚0.5322673655888507 /loss/FRt2/detection_loss‚1.5600393699288084 1loss/SRqqt2/detection_loss‚0.6105757817475976 %loss/total_loss‚7.794516013382272 epoch‚25 trainer/global_step‚10049 _step‚828 4loss/FBt2/assignment_loss‚0.00010660132189902406 _runtime‚8619.519937406 0loss/FRt1/assignment_loss‚1.5480367838174511 2loss/SRqqt2/assignment_loss‚0.3853421099256018 % _timestamp‚1.7843489608266504e+09¼$ Fä‘ò ‚(:† ¦‚ìÒäç”” gpu.0.gpu‚81.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚252.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žQïó ‚(:ä ¦‚ìÒàù­—# disk.dm-0.out‚26769.03369140625 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚26769.03369140625" disk.nvme0n1.in‚15742.67578125& disk.nvme0n1.out‚26769.03369140625 disk.dm-1.in‚ 0.1328125$ disk./.usageGB‚588.4499549865723 disk.sda.out‚268.9375 disk.sda3.out‚268.9375# memory_percent‚97.0569372177124 disk.sda.in‚ 0.1328125" proc.memory.rssMB‚ 2741.4765625 proc.cpu.threads‚17 disk.drbd2165.out‚0( proc.memory.availableMB‚ 482.19140625 network.recv‚ 2.351169e+06 disk.dm-0.in‚15742.67578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚15742.67578125 disk.sda1.out‚0 disk.dm-1.out‚268.9375) disk./.usagePercent‚39.49137022154769 disk.sda1.in‚0) proc.memory.percent‚16.73264503479004 cpu‚26.184436794732925 network.sent‚ 1.6071496e+07 disk.rbd1.out‚26.53271484375GšÖ|“ô ‚(:ˆ µ‚ìÒòèÖ” gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ÐÎ}øõ ‚(:í µ‚ìÒ„†á• disk.rbd0.in‚0 disk.dm-1.out‚ 269.4140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚26.53271484375 disk.sda1.in‚0$ memory_percent‚97.05827236175537) proc.memory.percent‚16.73264503479004 disk.sda2.in‚0# disk.dm-0.out‚26771.74951171875( proc.memory.availableMB‚ 481.97265625 network.sent‚ 1.6121692e+07 network.recv‚ 2.355882e+06* disk./.usagePercent‚39.488743584246386 disk.sda3.out‚ 269.4140625 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚26771.74951171875" proc.memory.rssMB‚ 2741.4765625# disk./.usageGB‚588.410816192627 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚15761.15234375 cpu‚26.18404123200887" disk.nvme0n1.in‚15761.15234375& disk.nvme0n1.out‚26771.74951171875$ disk.nvme0c0n1.in‚15761.15234375 disk.sda.in‚ 0.1328125 proc.cpu.threads‚17 disk.sda.out‚ 269.4140625 disk.sda2.out‚0P¼Éä’ö ‚(:‡ Ä‚ìÒ« Å“ gpu.0.gpu‚20.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚269.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.8552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÿuCù÷ ‚(:î Ä‚ìÒåÛ˜– disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚15794.61328125) proc.memory.percent‚16.73264503479004 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.out‚ 269.6640625 disk.sda3.out‚ 269.6640625( proc.memory.availableMB‚ 481.97265625" proc.memory.rssMB‚ 2741.4765625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26776.60888671875 disk.sda2.in‚0& disk.nvme0n1.out‚26776.60888671875 disk.dm-1.out‚ 269.6640625$ disk.nvme0c0n1.in‚15794.61328125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 network.recv‚ 2.35898e+06* disk./.usagePercent‚39.488752544510085$ disk./.usageGB‚588.4109497070312 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚26.183922139039613 network.sent‚ 1.6126558e+07 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚26.59521484375$ memory_percent‚97.05827236175537 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚26776.60888671875" disk.nvme0n1.in‚15794.613281254þg‡2ø - _wandbruntime‚8659  _runtime‚8659ýCK$Èù  _step‚829 ' lr-AdamW/pg1‚0.00022817332529007173 ' lr-AdamW/pg2‚0.00022817332529007173 trainer/global_step‚10099 # _timestamp‚1.78434900099082e+09 _runtime‚8659.684439352ÈÁÓÉÍú Ç ' lr-AdamW/pg1‚0.00022817332529007173 ' lr-AdamW/pg2‚0.00022817332529007173 trainer/global_step‚10099 # _timestamp‚1.78434900099082e+09 _runtime‚8659.684439352 _step‚829½@‰þÍ2û - _wandbruntime‚8659  _runtime‚8659©äÇX ü õO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80103710 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10668114126803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80103712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338152242505728 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546093802004992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348950895620608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10668454676693 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546094160383488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21718154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93564186 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348950992324096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Míê(ý úO _runtime‚8659.685647393 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10798749818368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1546093802004992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10668114126803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80103712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338152242505728 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546093802004992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348950895620608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10668454676693 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546094160383488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21718154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93564186 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348950992324096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80103710 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1535556630515712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1527656160747520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80111342 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1535556647555072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10685425145856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39415613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1546242414047232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93573252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784349001605967e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80111342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17695126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338185671264256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80111344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21720232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39415355 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21720487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348985347759104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10685425145856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1538325626478047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10669465730527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93563261 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1535556647555072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10669125180525 _runtime‚8660.299764093 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1535556630515712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10685766492160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173674603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80111344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1546242055661568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173674603 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684596 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10799781579776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10799695107072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338185652652032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1538325268888685 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚831 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1527656143708160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872é¼e®€!…N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80111342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17695126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338185671264256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80111344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21720232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39415355 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21720487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348985347759104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10685425145856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1538325626478047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10669465730527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93563261 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1535556647555072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10669125180525 _runtime‚8660.299764093 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1535556630515712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10685766492160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173674603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80111344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1546242055661568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173674603 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684596 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10799781579776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10799695107072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oveH¨-]rsize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338185652652032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1538325268888685 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚831 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1527656143708160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10685766492160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1535556630515712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1527656160747520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80111342 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1535556647555072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10685425145856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39415613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1546242414047232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93573252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784349001605967e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10735108020736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8700.745440495 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94018638 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚834 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10735449359872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39602139 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39601864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1553506548988416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10849906712064 trainer/global_step‚10149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1542771458007040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350673482052096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1545553897243599 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1534835166629888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10718730613711 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843490420517187e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80485312 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21821508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17780628 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339823680432128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1553506907366912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10849820232192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80485310 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94008661 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350673587144192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1553506907366912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80485312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10719071163601 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1542771440967680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0›%bª(!üO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94018638 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚834 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10735449359872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39602139 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39601864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1553506548988416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10849906712064 trainer/global_step‚10149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1542771458007040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350673482052096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1545553897243599 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1534835166629888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10718730613711 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843490420517187e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80485312 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21821508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17780628 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339823680432128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1553506907366912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10849820232192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80485310 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94008661 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350673587144192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1553506907366912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80485312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10719071163601 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1542771440967680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1542771458007040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10735449359872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94018638 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1545554254832849 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1534835183669248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174503950 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10735108020736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8700.745440495Âm™í‹2!- _wandbruntime‚8701  _runtime‚8701äÍñâ‘!€N SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1553655161030656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339857109190656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1542918699036672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17782373 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10719741667433 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10736120647680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10736120647680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174520648 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21823572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21823323 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1534981694341120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350708035771392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10736461993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80492944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚835 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80492942 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1545701418969193 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1542918699036672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39605696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174510655 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1553655161030656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94017713 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94027704 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1545701418969193 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1542918699036672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39605696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174510655 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1553655161030656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94017713 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94027704 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1553655161030656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339857109190656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1542918699036672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17782373 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10719741667433 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10736120647680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10736120647680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174520648 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21823572ÃîtÁ2“!- _wandbruntime‚8701  _runtime‚8701쨘“ ”!† 3loss/SRqqt1/assignment_loss‚0.47981489056411003 0loss/FRt1/assignment_loss‚1.5679338886943204 2loss/FBt1/assignment_loss‚0.001337912009220954 /loss/FRt2/detection_loss‚1.5039030634090487 1loss/SRqqt1/detection_loss‚0.8937387061045885 1loss/SRqqt2/detection_loss‚0.5785534264225151 epoch‚25 trainer/global_step‚10149 0loss/FRt2/assignment_loss‚0.5247159622171346 2loss/SRqqt2/assignment_loss‚0.4626354315545626 /loss/FRt1/detection_loss‚0.5767672977601271 .loss/FBt1/detection_loss‚0.524015596776735 /loss/FBt2/detection_loss‚0.5170824795948445 %loss/total_loss‚7.630574120827305 _runtime‚8701.364643284 _step‚836 3loss/FBt2/assignment_loss‚7.546572009741153e-05 % _timestamp‚1.7843490426713493e+09]… ;•!‹ trainer/global_step‚10149 0loss/FRt2/assignment_loss‚0.5247159622171346 2loss/SRqqt2/assignment_loss‚0.4626354315545626 /loss/FRt1/detection_loss‚0.5767672977601271 .loss/FBt1/detection_loss‚0.524015596776735 /loss/FBt2/detection_loss‚0.5170824795948445 %loss/total_loss‚7.630574120827305 _runtime‚8701.364643284 _step‚836 3loss/FBt2/assignment_loss‚7.546572009741153e-05 % _timestamp‚1.7843490426713493e+09 3loss/SRqqt1/assignment_loss‚0.47981489056411003 0loss/FRt1/assignment_loss‚1.5679338886943204 2loss/FBt1/assignment_loss‚0.001337912009220954 /loss/FRt2/detection_loss‚1.5039030634090487 1loss/SRqqt1/detection_loss‚0.8937387061045885 1loss/SRqqt2/detection_loss‚0.5785534264225151 epoch‚25ű=’“–!‚(:ˆ €ƒìÒ—¨¬“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.217' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.2434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s\5éñ—!‚(:æ €ƒìÒî¢Í– disk.rbd0.out‚0 disk.dm-1.out‚271.0625$ disk.nvme0c0n1.in‚15848.54296875" proc.memory.rssMB‚ 2741.4765625 network.recv‚ 2.373733e+06$ disk./.usageGB‚588.4105339050293 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚15848.54296875) proc.memory.percent‚16.73264503479004 network.sent‚ 1.6204694e+07 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚27.14208984375 disk.dm-0.in‚15848.54296875 disk.sda.out‚271.0625 disk.sda3.out‚271.0625$ memory_percent‚97.05970287322998 disk.sda1.in‚0( disk.nvme0c0n1.out‚26795.11279296875 disk.sda2.out‚0* disk./.usagePercent‚39.488724639688854 disk.drbd2165.out‚0 cpu‚26.182485926979727 proc.cpu.threads‚17 disk.rbd0.in‚0# disk.dm-0.out‚26795.11279296875( proc.memory.availableMB‚ 481.73828125 disk.dm-1.in‚ 0.1328125& disk.nvme0n1.out‚26795.11279296875 disk.sda3.in‚ 0.1328125Ïé­˜!‚(:’ ƒìÒÃÁ“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.954' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.19080000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hy½Áú™!‚(:ï ƒìÒÚ£€– disk.sda3.in‚ 0.1328125 disk.sda1.out‚0# disk.dm-0.out‚26798.19873046875 disk.sda2.out‚0$ memory_percent‚97.05817699432373 proc.cpu.threads‚17$ disk./.usageGB‚588.4103584289551 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚15848.54296875" disk.nvme0n1.in‚15848.54296875 disk.sda3.out‚ 271.34765625( disk.nvme0c0n1.out‚26798.19873046875 disk.dm-0.in‚15848.54296875& disk.nvme0n1.out‚26798.19873046875 network.sent‚ 1.626771e+07 disk.rbd0.out‚0 disk.dm-1.out‚ 271.34765625 disk.drbd2165.in‚0 disk.rbd1.out‚27.14990234375( proc.memory.availableMB‚ 481.98828125) proc.memory.percent‚16.73264503479004) disk./.usagePercent‚39.48871286334228 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125" proc.memory.rssMB‚ 2741.4765625 network.recv‚ 2.378202e+06 disk.sda.out‚ 271.34765625 disk.sda2.in‚0 cpu‚26.18223457833612 disk.rbd0.in‚0 disk.drbd2165.out‚0«m^™2š!- _wandbruntime‚8741  _runtime‚8741¿ïŠ¤Ê›!Ä ' lr-AdamW/pg1‚0.00022457446155382587 ' lr-AdamW/pg2‚0.00022457446155382587 trainer/global_step‚10199 % _timestamp‚1.7843490828707619e+09 _runtime‚8741.564278214 _step‚837é8¢ Ïœ!É ' lr-AdamW/pg1‚0.00022457446155382587 ' lr-AdamW/pg2‚0.00022457446155382587 trainer/global_step‚10199 % _timestamp‚1.7843490828707619e+09 _runtime‚8741.564278214 _step‚837ÅP¬4ƒ2!- _wandbruntime‚8741  _runtime‚8741‘M(#ž!÷O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39792063 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17867881 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10900886963200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1560919654350336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94463113 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352395986709504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17867878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175340002 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80866912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560919295971840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175330023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21923919 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10769687650509 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 trainer/global_step‚10199 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1550133509488640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10900975540224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341495099746304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10785803522560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21924182 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560919295971840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_poolÓ ¡Õ.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80866910 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94473090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80866912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39791797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352396093898752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224@ Å(Ÿ!üO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94463113 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352395986709504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17867878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175340002 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80866912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560919295971840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175330023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21923919 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10769687650509 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 trainer/global_step‚10199 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1550133509488640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10900975540224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341495099746304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10785803522560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21924182 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560919295971840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80866910 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94473090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80866912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39791797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352396093898752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39792063 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17867881 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10900886963200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1560919654350336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Æd&i¹2 !- _wandbruntime‚8742  _runtime‚87423k‘û'¡!N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175346707 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356700 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80874542 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94482156 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94472165 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚839 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94482156 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1550280750518272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10786816149504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39795602 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352430426922496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8742.182275919 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80874542 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21926243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1561067908014080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21925979 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10770698704343 trainer/global_step‚10199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10901898417664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175356700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80874544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94472165 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550280733478912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80874544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843490834884593e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10787157495808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17869623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1542307210895360Éÿ½ '¢!†N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10787157495808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17869623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1542307210895360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175346707 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356700 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80874542 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94482156 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94472165 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚839 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94482156 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1550280750518272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10786816149504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39795602 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352430426922496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8742.182275919 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80874542 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21926243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1561067908014080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21925979 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10770698704343 trainer/global_step‚10199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10901898417664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175356700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80874544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94472165 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550280733478912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80874544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843490834884593e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10787157495808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚10249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1568332401333760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10836840363520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176054 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39982154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22027023 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1568332401333760 _step‚842 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10952134125568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1549486233817088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1557495560970240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22026759 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1568332042955264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343166537672704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94927542 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10836499024384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94917565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176166075 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354118690410496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1560306197404615 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8782.545691284 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1557495560970240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94927542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343166556284928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354118583221248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10836499024384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17955128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1568332042955264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10819963587527 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81248512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843491238519475e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1557495543930880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81248510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1560306554993865 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10836840363520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1557495543930880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39981887 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81248512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10952134125568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1549486233817088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1557495560970240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22026759 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1568332042955264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343166537672704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94927542 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10836499024384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94917565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176166075 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354118690410496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1560306197404615 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8782.545691284 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1557495560970240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94927542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343166556284928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354118583221248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10836499024384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17955128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1568332042955264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10819963587527 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81248512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843491238519475e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1557495543930880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81248510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1560306554993865 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10836840363520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1557495543930880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39981887 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81248512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚10249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1568332401333760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10836840363520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176054 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39982154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22027023 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ÊoëÉm2²!- _wandbruntime‚8783  _runtime‚8783¿ãÅ¿H³!N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81256144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1568480654997504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17956873 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10953057682944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1568480654997504 _step‚843 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10837511651328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1557642801999872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1557642784960512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39985585 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1557642801999872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560454076719571 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10249 _runtime‚8783.160372531 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354153024114176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176192752 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1549632761528320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10837852997632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354153137587712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343199985043456 % _timestamp‚1.7843491244664743e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94936608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820974641249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22028712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94936608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549632744488960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1568480296611840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10837511651328Q÷r{ '´!†N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10837852997632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354153137587712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343199985043456 % _timestamp‚1.7843491244664743e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94936608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820974641249 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22028712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94936608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549632744488960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1568480296611840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10837511651328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81256144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1568480654997504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17956873 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10953057682944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1568480654997504 _step‚843 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10837511651328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1557642801999872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1557642784960512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39985585 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1557642801999872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1560454076719571 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10249 _runtime‚8783.160372531 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354153024114176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176192752 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1549632761528320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976Ë”*£w2µ!- _wandbruntime‚8783  _runtime‚8783Ü‚øÑ ¶!† 2loss/SRqqt1/assignment_loss‚0.5603651566129337 3loss/SRqqt2/assignment_loss‚0.46639364995204463 1loss/SRqqt2/detection_loss‚0.5606731467587239 /loss/FBt1/detection_loss‚0.5035206280415878 trainer/global_step‚10249 2loss/FBt1/assignment_loss‚0.014518665475267057 /loss/FRt1/detection_loss‚0.5521765530279866 /loss/FRt2/detection_loss‚1.4766300368713572 /loss/FBt2/detection_loss‚0.5422394789906869 1loss/SRqqt1/detection_loss‚0.7290350006640489 epoch‚25 $ _timestamp‚1.784349124469546e+09 0loss/FRt1/assignment_loss‚1.5100424771713317 0loss/FRt2/assignment_loss‚0.6759809037675003 3loss/FBt2/assignment_loss‚9.314422562921112e-05 %loss/total_loss‚7.591668841559097 _runtime‚8783.162838356 _step‚844ƒz ·!‹ 2loss/SRqqt1/assignment_loss‚0.5603651566129337 3loss/SRqqt2/assignment_loss‚0.46639364995204463 1loss/SRqqt2/detection_loss‚0.5606731467587239 /loss/FBt1/detection_loss‚0.5035206280415878 trainer/global_step‚10249 2loss/FBt1/assignment_loss‚0.014518665475267057 /loss/FRt1/detection_loss‚0.5521765530279866 /loss/FRt2/detection_loss‚1.4766300368713572 /loss/FBt2/detection_loss‚0.5422394789906869 1loss/SRqqt1/detection_loss‚0.7290350006640489 epoch‚25 $ _timestamp‚1.784349124469546e+09 0loss/FRt1/assignment_loss‚1.5100424771713317 0loss/FRt2/assignment_loss‚0.6759809037675003 3loss/FBt2/assignment_loss‚9.314422562921112e-05 %loss/total_loss‚7.591668841559097 _runtime‚8783.162838356 _step‚844Ìf 6œ¸!‚(:‘ ˃ìÒüûö“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.335' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.266999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tnæÜó¹!‚(:è ˃ìÒÜšî– disk.dm-0.in‚15910.16796875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 272.39453125$ memory_percent‚97.05965518951416" proc.memory.rssMB‚ 2741.4765625 network.recv‚ 2.392789e+06 disk.sda3.in‚ 0.1328125( proc.memory.availableMB‚ 481.74609375) proc.memory.percent‚16.73264503479004* disk./.usagePercent‚39.488755872608024 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚15910.16796875 disk.rbd1.out‚32.3125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚26814.3037109375 cpu‚26.181045753672592 network.sent‚ 1.6345986e+07 disk.sda2.in‚0 disk.sda3.out‚ 272.39453125$ disk.nvme0c0n1.in‚15910.16796875% disk.nvme0n1.out‚26814.3037109375$ disk./.usageGB‚588.4109992980957" disk.dm-0.out‚26814.3037109375 disk.sda.out‚ 272.39453125 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.drbd2165.in‚0.¶oãº!‚(:’ ÚƒìÒœ¼º” gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.11' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.422000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?–.Dó»!‚(:è ÚƒìÒ²Äá–# disk.dm-0.out‚26818.05322265625 disk.sda2.out‚0) disk./.usagePercent‚39.48849756100598 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.in‚ 2.01953125 network.sent‚ 1.6409137e+07 network.recv‚ 2.396696e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.out‚ 272.73046875" disk.nvme0n1.in‚15910.16796875 disk.sda2.in‚0& disk.nvme0n1.out‚26818.05322265625# memory_percent‚97.0592737197876( proc.memory.availableMB‚ 481.80859375" proc.memory.rssMB‚ 2741.4765625) proc.memory.percent‚16.73264503479004 cpu‚26.180795587056817 proc.cpu.threads‚17 disk.dm-1.out‚ 272.73046875 disk.sda.in‚ 0.1328125 disk.rbd1.out‚32.375$ disk./.usageGB‚588.4071502685547 disk.drbd2165.out‚0 disk.dm-0.in‚15910.16796875 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 272.73046875$ disk.nvme0c0n1.in‚15910.16796875( disk.nvme0c0n1.out‚26818.05322265625ÆÕ‚“¼!‚(:ˆ éƒìÒýÂå“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðºîí½!‚(:â éƒìÒ»ø –$ disk./.usageGB‚588.4071884155273" disk.dm-0.out‚26820.9560546875% disk.nvme0n1.out‚26820.9560546875" proc.memory.rssMB‚ 2741.4765625 network.recv‚ 2.399975e+06' disk.nvme0c0n1.out‚26820.9560546875) disk./.usagePercent‚39.48850012108132 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 273.2109375 disk.sda1.out‚0 disk.dm-0.in‚15910.16796875 disk.sda.out‚ 273.2109375" disk.nvme0n1.in‚15910.16796875' proc.memory.availableMB‚ 481.8203125) proc.memory.percent‚16.73264503479004 disk.rbd1.out‚32.375 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 273.2109375 disk.drbd2165.in‚0 network.sent‚ 1.6414276e+07 disk.drbd2165.out‚0 disk.sda2.out‚0 cpu‚26.180379125260394 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚15910.16796875$ memory_percent‚97.05920219421387Âp2¾!- _wandbruntime‚8823  _runtime‚8823xæÊ¿!Ä _step‚845 ' lr-AdamW/pg1‚0.00022097558857138883 ' lr-AdamW/pg2‚0.00022097558857138883 trainer/global_step‚10299 % _timestamp‚1.7843491648347628e+09 _runtime‚8823.528384456 _ËÏÀ!É ' lr-AdamW/pg1‚0.00022097558857138883 ' lr-AdamW/pg2‚0.00022097558857138883 trainer/global_step‚10299 % _timestamp‚1.7843491648347628e+09 _runtime‚8823.528384456 _step‚845Í:’¶2Á!- _wandbruntime‚8823  _runtime‚88232Óß ü'Â!öO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81630112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95372017 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564857612451840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177012106 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95372017 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355841246898176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10870920624325 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10887194526208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10870580074435 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40172086 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11003271299072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22129708 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81630112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95381994 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1575745148317184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564857595412480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95381994 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344837994211328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚846 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81630110 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1575745148317184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355841349893120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1575744789938688 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1556811784450048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10887535865344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784349164838517e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042378 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22129967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1575744789938688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10887535865344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088@¯‡ê× Ã!ûO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10870580074435 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40172086 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11003271299072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22129708 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81630112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95381994 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1575745148317184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564857595412480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95381994 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344837994211328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚846 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81630110 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1575745148317184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355841349893120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1575744789938688 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1556811784450048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10887535865344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784349164838517e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042378 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22129967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1575744789938688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10887535865344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81630112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95372017 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564857612451840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177012106 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95372017 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355841246898176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10870920624325 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10887194526208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0κ2Ä!- _wandbruntime‚8824  _runtime‚8824÷Íž'Å!N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95391060 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚8824.149012526 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028804 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344871404357632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1567829869210717 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11004296442368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10871591128157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10888207153152 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1575893043595264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565004836442112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10888207153152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344871422969856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567830226800079 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1556958278082560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565004853481472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355875803787776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1565004853481472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565004836442112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22132041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1575893401980928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11004380817920 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40175916 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10888548499456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚10299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177028804 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1575893401980928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚847 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81637744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177018811 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81637742 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177018811 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565004853481472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355875803787776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1565004853481472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1565004836442112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22132041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1575893401980928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11004380817920 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40175916 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10888548499456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚10299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177028804 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1575893401980928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚847 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81637744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177018811 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81637742 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177018811 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚8824.149012526 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028804 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344871404357632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1567829869210717 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11004296442368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10871591128157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10888207153152 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1575893043595264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565004836442112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10888207153152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344871422969856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567830226800079 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1556958278082560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357563841374720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8864.325493964 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95836446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18129628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82011710 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22232033 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1583157536922112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1583157895300608 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22232293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1572219663933440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚850 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82011710 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10937890028032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848158 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40361661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82011712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22232293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1572219663933440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚850 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82011710 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10937890028032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848158 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40361661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82011712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564137318043648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357563841374720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8864.325493964 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95836446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18129628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82011710 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22232033 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1583157536922112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1583157895300608 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888Òx3ø2Ô!- _wandbruntime‚8864  _runtime‚8864Üȵ'Õ!N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40365471 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11055538887680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938902654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95835521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82019342 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1572366904963072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1564283828715520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22234098 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1575206376880587 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10938902654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357598296796160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10922207615065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95835521 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚851 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1572366887923712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95845512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10922548165067 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1583306148964352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1583306148964352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1572366904963072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82019342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22234350 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11055454512128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95845512 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1572366887923712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18131376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438ýRÔu Ö!†N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1583306148964352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1583306148964352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1572366904963072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82019342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22234350 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11055454512128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95845512 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1572366887923712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18131376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40365471 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11055538887680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938902654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95835521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82019342 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1572366904963072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1564283828715520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22234098 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1575206376880587 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10938902654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357598296796160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10922207615065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95835521 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚851 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1572366887923712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95845512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10922548165067Ó úª²2×!- _wandbruntime‚8864  _runtime‚8864µîW Ø!† %loss/total_loss‚7.681615634810965 epoch‚25 2loss/SRqqt1/assignment_loss‚0.5899715122446012 3loss/SRqqt2/assignment_loss‚0.41865414730783457 4loss/FBt1/assignment_loss‚1.4074494211298907e-05 1loss/SRqqt1/detection_loss‚0.7709325418140538 _runtime‚8864.944671382 0loss/FRt2/assignment_loss‚0.6450830147454613 /loss/FRt1/detection_loss‚0.5828443458172708 trainer/global_step‚10349 /loss/FRt2/detection_loss‚1.5514958034490087 -loss/SRqqt2/detection_loss‚0.712564235721 0loss/FBt1/detection_loss‚0.45154159290733625 /loss/FBt2/detection_loss‚0.4269231711682582 _step‚852 0loss/FRt1/assignment_loss‚1.5307930827140808 3loss/FBt2/assignment_loss‚0.0007981124278487943 % _timestamp‚1.7843492062513723e+09¾ˆìâÙ!‹ 0loss/FRt2/assignment_loss‚0.6450830147454613 /loss/FRt1/detection_loss‚0.5828443458172708 trainer/global_step‚10349 /loss/FRt2/detection_loss‚1.5514958034490087 -loss/SRqqt2/detection_loss‚0.712564235721 0loss/FBt1/detection_loss‚0.45154159290733625 /loss/FBt2/detection_loss‚0.4269231711682582 _step‚852 0loss/FRt1/assignment_loss‚1.5307930827140808 3loss/FBt2/assignment_loss‚0.0007981124278487943 % _timestamp‚1.7843492062513723e+09 %loss/total_loss‚7.681615634810965 epoch‚25 2loss/SRqqt1/assignment_loss‚0.5899715122446012 3loss/SRqqt2/assignment_loss‚0.41865414730783457 4loss/FBt1/assignment_loss‚1.4074494211298907e-05 1loss/SRqqt1/detection_loss‚0.7709325418140538 _runtime‚8864.944671382Ôò&Nº“Ú!‚(:ˆ –„ìÒ¸¹À– gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.343' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.2686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú…«\ôÛ!‚(:é –„ìÒ³ïõ— network.recv‚ 2.411353e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 15965.1015625 network.sent‚ 1.648725e+07 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚26.17928758991051 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda.out‚ 274.10546875 disk.sda3.out‚ 274.10546875$ memory_percent‚97.05755710601807( proc.memory.availableMB‚ 482.08984375" proc.memory.rssMB‚ 2741.4765625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 274.10546875' disk.nvme0c0n1.out‚26831.2626953125) proc.memory.percent‚16.73264503479004! disk.nvme0n1.in‚ 15965.1015625 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚32.66455078125) disk./.usagePercent‚39.48878710552721 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 15965.1015625 disk.sda.in‚ 0.1328125$ disk./.usageGB‚588.4114646911621" disk.dm-0.out‚26831.2626953125 disk.sda2.out‚0% disk.nvme0n1.out‚26831.2626953125%!lžÜ!‚(:“ ¥„ìÒÎøª“ gpu.0.gpu‚41.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éo¨aîÝ!‚(:ã ¥„ìÒ–) proc.memory.percent‚16.73264503479004$ disk./.usageGB‚588.4113235473633 disk.dm-0.out‚26834.58203125 disk.sda3.out‚ 274.46484375 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 15965.1015625 disk.sda3.in‚ 0.1328125# disk.nvme0n1.out‚26834.58203125" proc.memory.rssMB‚ 2741.4765625 disk.dm-0.in‚ 15965.1015625 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 15965.1015625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.out‚32.66455078125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 274.46484375' proc.memory.availableMB‚ 481.8828125 cpu‚26.179052432672925 network.sent‚ 1.6549319e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.415199e+06) disk./.usagePercent‚39.48877763324844% disk.nvme0c0n1.out‚26834.58203125 disk.sda.out‚ 274.46484375 disk.rbd0.in‚0# memory_percent‚97.0588207244873Qgw“Þ!‚(:ˆ ´„ìÒ´ÝÁ“ gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µªJºêß!‚(:ß ´„ìÒ“Òƒ–" disk.nvme0c0n1.in‚ 16027.421875 disk.dm-0.in‚ 16027.421875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚26840.923828125 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0! disk.dm-0.out‚26840.923828125 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda.out‚ 274.890625( proc.memory.availableMB‚ 481.55078125) disk./.usagePercent‚39.48849551294571 disk.rbd0.out‚0 disk.dm-1.out‚ 274.890625 disk.sda3.out‚ 274.890625 disk.sda1.out‚0 disk.nvme0n1.in‚ 16027.421875$ disk.nvme0n1.out‚26840.923828125) proc.memory.percent‚16.73264503479004 disk.dm-1.in‚ 0.1328125 network.sent‚ 1.6554506e+07$ disk./.usageGB‚588.4071197509766 disk.rbd1.in‚ 2.01953125" proc.memory.rssMB‚ 2741.4765625 cpu‚26.178892534091812 disk.rbd1.out‚32.82080078125$ memory_percent‚97.06084728240967 network.recv‚ 2.418549e+06Vþœ~2à!- _wandbruntime‚8905  _runtime‚8905 6y‰Æá!À % lr-AdamW/pg1‚0.000217377630923461 % lr-AdamW/pg2‚0.000217377630923461 trainer/global_step‚10399 % _timestamp‚1.7843492466507726e+09 _runtime‚8905.347507199 _step‚853ùÍËâ!Å _runtime‚8905.347507199 _step‚853 % lr-AdamW/pg1‚0.000217377630923461 % lr-AdamW/pg2‚0.000217377630923461 trainer/global_step‚10399 % _timestamp‚1.7843492466507726e+09Õ"‰42ã!- _wandbruntime‚8905  _runtime‚8905 ïVý'ä!÷O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10988926868992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82393310 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1590570283905536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18216881 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96280921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18216878 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10988585529856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96280921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1579581698375680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1579581715415040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82393312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1579581715415040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40552575 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348180851451904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1571462834597888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843492466547487e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1579581698375680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1571462851637248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1590570642284032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82393312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11105535929856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348180870064128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10971813048251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674231 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1590570642284032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96290898 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10988585529856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚854 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1590570283905536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96290898 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11105620312576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚10399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22335445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82393310 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40552323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359286490376704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359286387381760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚8905.348553133 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10972153598141 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22335694 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10988926868992kÿ¨Ù("å!üO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22335694 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10988926868992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10988926868992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82393310 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1590570283905536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18216881 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96280921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18216878 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10988585529856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96280921 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1579581698375680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1579581715415040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82393312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1579581715415040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40552575 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348180851451904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1571462834597888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843492466547487e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1579581698375680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1571462851637248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1590570642284032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82393312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11105535929856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348180870064128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10971813048251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674231 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1590570642284032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96290898 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10988585529856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚854 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1590570283905536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96290898 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11105620312576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚10399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_b†‡Úatch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22335445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82393310 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40552323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359286490376704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359286387381760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚8905.348553133 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10972153598141 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169Ö¾®ë2æ!- _wandbruntime‚8905  _runtime‚8905`^’~'ç!N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40556108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348214280210432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579728956444672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178690915 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1579728956444672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10989598156800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22337745 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22337485 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10989598156800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚855 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178700908 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82400942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348214298822656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82400944 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1590718537562112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18218623 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1590718895947776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10989939503104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96299964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18218626 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96289973 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579728939405312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154^Y}± 'è!†N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1571609362309120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82400942 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10973164651975 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1571609345269760 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178690915 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40556108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348214280210432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579728956444672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178690915 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1579728956444672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10989598156800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22337745 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22337485 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10989598156800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚855 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178700908 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82400942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348214298822656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82400944 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1590718537562112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18218623 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1590718895947776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10989939503104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96299964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18218626 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96289973 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579728939405312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11161849647760391 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1685231328010559 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029775764793157578 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.043454937636852264 *jet/accuracy_2_of_4‚0.7472164630889893 +particle/specificity‚0.9015378355979919 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953631043434143 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6226109266281128 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784349265849225e+09 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2323226034641266 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6597087383270264 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21988938748836517 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6789070963859558 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21590782701969147 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8863815069198608 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_3_of_3‚0.054812975227832794 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5908951759338379 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028257863596081734 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28713953495025635 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.499399334192276 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29802265763282776 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960177540779114 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28713953495025635 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.34560853242874146 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29101499915122986 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6711514592170715 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988865852355957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02393481507897377 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3185315728187561 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3364816904067993 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7552098035812378 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24340608716011047 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7363224625587463 )jet/accuracy_2_of_3‚0.533685564994812 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22917978465557098 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21617117524147034 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6759240031242371 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09445052593946457 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7239016890525818 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34673523902893066 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.38100796937942505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7789109349250793 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6463260054588318 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5311377644538879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7792625427246094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2906157970428467 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_4_of_4‚0.01628045178949833 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7527645826339722 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7780551910400391 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21249312162399292 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08259724080562592 particle/accuracy_1_of_3‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7253322601318359 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6216800212860107 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_1‚0.3489666283130646 )jR(å„áet/accuracy_1_of_2‚0.771885097026825 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49895399808883667 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_2‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4490741193294525 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7956727147102356 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8696901202201843 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06914519518613815 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6341195702552795 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8328392505645752 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8612055778503418 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6526485681533813 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.06128383427858353 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.948907732963562 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8832955360412598 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38172945380210876 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6455904245376587 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2298526018857956 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 _runtime‚8924.543580517 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6971685290336609 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5158343315124512 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23122736811637878 (particle/accuracy‚0.8396654725074768 trainer/global_step‚10399 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24191994965076447 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6788180470466614 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11288901418447495 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23680269718170166 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6450173258781433 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2657996714115143 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29132241010665894 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.872826874256134 +jet/accuracy_2_of_2‚0.14122910797595978 *jet/accuracy_1_of_4‚0.9799431562423706 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28591418266296387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6570082902908325 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7053437829017639 particle/accuracy_1_of_1‚1 5validation_average_jet_accuracy‚0.381703644990921 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27072566747665405 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02393481507897377 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8587160706520081 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23266223073005676 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8587160706520081 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9354465007781982 particle/accuracy_4_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5973691344261169 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027995910495519638 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6708178520202637 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2984992563724518 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9107396006584167 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6308378577232361 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6743987202644348 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37649887800216675 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7135045528411865 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7823169231414795 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833134651184 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2778010964393616 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 _step‚857 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8182366490364075 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13014745712280273 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029351728036999702 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7107633352279663 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03633350878953934 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6463260054588318 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7209508419036865 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6971859335899353 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154/!øw=ð!ñz particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8696901202201843 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06914519518613815 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6341195702552795 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8328392505645752 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8612055778503418 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6526485681533813 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.06128383427858353 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.948907732963562 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8832955360412598 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38172945380210876 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6455904245376587 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2298526018857956 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 _runtime‚8924.543580517 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6971685290336609 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5158343315124512 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23122736811637878 (particle/accuracy‚0.8396654725074768 trainer/global_step‚10399 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24191994965076447 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6788180470466614 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11288901418447495 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23680269718170166 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6450173258781433 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2657996714115143 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29132241010665894 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.872826874256134 +jet/accuracy_2_of_2‚0.14122910797595978 *jet/accuracy_1_of_4‚0.9799431562423706 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28591418266296387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6570082902908325 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7053437829017639 particle/accuracy_1_of_1‚1 5validation_average_jet_accuracy‚0.381703644990921 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27072566747665405 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02393481507897377 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8587160706520081 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23266223073005676 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8587160706520081 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9354465007781982 particle/accuracy_4_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5973691344261169 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027995910495519638 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6708178520202637 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2984992563724518 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9107396006584167 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6308378577232361 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6743987202644348 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37649887800216675 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7135045528411865 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7823169231414795 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833134651184 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2778010964393616 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 _step‚857 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8182366490364075 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13014745712280273 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029351728036999702 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7107633352279663 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03633350878953934 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6463260054588318 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7209508419036865 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6971859335899353 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7366394996643066 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03668737784028053 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7611737847328186 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5616666078567505 +particle/sensitivity‚0.6134559512138367 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984343647956848 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966352581977844 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7728550434112549 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_3_of_3‚1 )loss/val_total_loss‚7.732738021020393 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0422038808465004 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.872826874256134 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969152808189392 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11161849647760391 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1685231328010559 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029775764793157578 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.043454937636852264 *jet/accuracy_2_of_4‚0.7472164630889893 +particle/specificity‚0.9015378355979919 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6953631043434143 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6226109266281128 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 $ _timestamp‚1.784349265849225e+09 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2323226034641266 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6597087383270264 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21988938748836517 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6789070963859558 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21590782701969147 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8863815069198608 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_3_of_3‚0.054812975227832794 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5908951759338379 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028257863596081734 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28713953495025635 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.499399334192276 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29802265763282776 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960177540779114 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28713953495025635 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.34560853242874146 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29101499915122986 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6711514592170715 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988865852355957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02393481507897377 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3185315728187561 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3364816904067993 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7552098035812378 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24340608716011047 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7363224625587463 )jet/accuracy_2_of_3‚0.533685564994812 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22917978465557098 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21617117524147034 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6759240031242371 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09445052593946457 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7239016890525818 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34673523902893066 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.38100796937942505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7789109349250793 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6463260054588318 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5311377644538879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7792625427246094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2906157970428467 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_4_of_4‚0.01628045178949833 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7527645826339722 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7780551910400391 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21249312162399292 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08259724080562592 particle/accuracy_1_of_3‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7253322601318359 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6216800212860107 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_1‚0.3489666283130646 )jet/accuracy_1_of_2‚0.771885097026825 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49895399808883667 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_2‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4490741193294525 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7956727147102356ÙE4€7‘ñ!‚(:† Ò„ìÒœ†¿“ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L‡ÿòò!‚(:ç Ò„ìÒËŃ– disk.sda.out‚ 275.78515625# memory_percent‚97.0785140991211* proc.memory.percent‚16.748762130737305" disk.nvme0c0n1.in‚ 16027.421875 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 16027.421875 disk.dm-0.in‚ 16027.421875% proc.memory.availableMB‚ 478.65625 cpu‚26.183709891858246( disk.nvme0c0n1.out‚26852.99755859375 network.sent‚ 1.6622366e+07 disk.dm-1.in‚ 0.1328125 disk.rbd0.in‚0# disk.dm-0.out‚26852.99755859375 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125) disk./.usagePercent‚39.48875766466077$ disk./.usageGB‚588.4110260009766& disk.nvme0n1.out‚26852.99755859375 disk.rbd1.out‚32.88330078125" proc.memory.rssMB‚ 2744.1171875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 275.78515625 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.recv‚ 2.42639e+06 disk.sda1.in‚0 disk.dm-1.out‚ 275.78515625™øýûó!‚(:’ á„ìÒ¯çÊ“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ?s¶ðô!‚(:å á„ìÒÝÁá– disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 16027.421875$ memory_percent‚97.07918167114258* proc.memory.percent‚16.748785972595215 cpu‚26.183398303530186 disk.sda2.in‚0$ disk./.usageGB‚588.4109916687012 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 276.07421875 disk.sda.out‚ 276.07421875 disk.nvme0n1.in‚ 16027.421875 disk.sda2.out‚0& proc.memory.availableMB‚ 478.546875) disk./.usagePercent‚39.48875536059296 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 network.sent‚ 1.6679589e+07 network.recv‚ 2.430987e+06 disk.sda3.out‚ 276.07421875 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚32.88330078125 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚26856.103515625! disk.dm-0.out‚26856.103515625 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 16027.421875# proc.memory.rssMB‚ 2744.12109375 proc.cpu.threads‚17$ disk.nvme0n1.out‚26856.103515625õ!‚(:’ ð„ìÒ…û®“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|äékùö!‚(:î ð„ìÒÎì½– disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 cpu‚26.183010251136267 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚16089.41015625 disk.sda3.out‚ 276.7109375 network.recv‚ 2.434308e+06 disk.sda.out‚ 276.7109375& disk.nvme0n1.out‚26861.92626953125 disk.rbd0.out‚0& proc.memory.availableMB‚ 478.546875 network.sent‚ 1.6684565e+07 disk.rbd1.in‚ 2.01953125 disk.dm-0.in‚16089.41015625# disk.dm-0.out‚26861.92626953125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.07918167114258) disk./.usagePercent‚39.48872617573406$ disk./.usageGB‚588.4105567932129 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚16089.41015625( disk.nvme0c0n1.out‚26861.92626953125 disk.rbd1.out‚32.94580078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 276.7109375 disk.sda2.out‚0# proc.memory.rssMB‚ 2744.12109375* proc.memory.percent‚16.748785972595215 disk.sda2.in‚0@JNN2÷!- _wandbruntime‚8964  _runtime‚8964eêÌ:Çø!Á & lr-AdamW/pg1‚0.0002155792842236001 & lr-AdamW/pg2‚0.0002155792842236001 trainer/global_step‚10449 $ _timestamp‚1.784349306128711e+09 _runtime‚8964.822300006 _step‚858›¥Ìù!Æ & lr-AdamW/pg1‚0.0002155792842236001 & lr-AdamW/pg2‚0.0002155792842236001 trainer/global_step‚10449 $ _timestamp‚1.784349306128711e+09 _runtime‚8964.822300006 _step‚858Ú¾02ú!- _wandbruntime‚8964  _runtime‚8964ùì4¿°û!÷O P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8964.823187933 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82895578 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22470948 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1591737180665956 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1599930330744832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18306648 _step‚859 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349868201478144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1588873316507648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1599930689123328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895576 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1591737538255206 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843493061294665e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11057014237184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96815672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11057014237184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11174924860928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1580697048444928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82895576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1580697031405568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18306651 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11175007146496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11040149260388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_;GMtrain_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1588873316507648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11057355576320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11057355576320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1599930689123328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96815672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96805695 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1599930330744832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1588873333547008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96805695 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚10449 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11040489810278 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179711250 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22470948 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1591737180665956 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1599930330744832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18306648 _step‚859 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349868201478144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1588873316507648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1599930689123328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895576 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1591737538255206 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843493061294665e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11057014237184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96815672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11057014237184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11174924860928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1580697048444928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82895576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1580697031405568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18306651 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11175007146496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11040149260388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1588873316507648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11057355576320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11057355576320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1599930689123328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96815672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96805695 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1599930330744832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1588873333547008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96805695 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚10449 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11040489810278 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179711250 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11058026864128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1589020557537280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1589020574576640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11058368210432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349901630236672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82903210 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82903208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1589020557537280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96814747 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40781419 _runtime‚8965.627067246 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11058026864128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179727948 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18308396 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1600078942787072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96824738 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40781172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349901648848896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727948 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843493069332464e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179717955 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361077586524160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚860 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96824738 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1589020574576640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22472779 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1591884702391550 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11058368210432 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96814747 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1600078584401408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361077685317632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1591885059980912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179717955 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82903208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11175956287488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1580843542077440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11041500864112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360]¹ hVÿ!…N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1580843559116800 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11058026864128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1589020557537280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1589020574576640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11058368210432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349901630236672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82903210 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82903208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1589020557537280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96814747 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40781419 _runtime‚8965.627067246 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11058026864128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179727948 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18308396 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1600078942787072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96824738 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40781172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349901648848896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727948 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843493069332464e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179717955 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361077586524160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚860 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96824738 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1589020574576640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22472779 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1591884702391550 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11058368210432 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96814747 T?DeviceStatsMonitor.on_train_batch_end/allWôjQµ ocated_bytes.all.freed‚1600078584401408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361077685317632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1591885059980912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179717955 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82903208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11175956287488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1580843542077440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11041500864112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82903210 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18308393 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1599113688335714 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11107709739008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180537323 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83277176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1588022564999168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83277178 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18393898 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351539658016768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚9005.611867784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22575145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596235367989248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607343077728256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180547302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1599113688335714 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11107709739008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180537323 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83277176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1588022564999168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83277178 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18393898 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351539658016768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚9005.611867784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22575145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596235367989248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607343077728256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180547302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83284810 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11091776801018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11108722365952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180564000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180554007 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180554007 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18395646 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytVM5[es.large_pool.allocated‚1588169092710400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97269199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362800446548992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362800549536768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1607491331384832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97279190 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11227462761472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180564000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22577231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83284810 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40972611 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596382609018880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97269199 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1596382626058240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596382609018880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚9006.228754252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83284808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1588169075671040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11227378385920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269=©Ã* '‘"†N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97269199 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1596382626058240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596382609018880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚9006.228754252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83284808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1588169075671040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11227378385920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11108722365952 trainer/global_step‚10499 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351573086775296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83284808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843493475349884e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83284810 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11091776801018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11108722365952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180564000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180554007 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180554007 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18395646 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1588169092710400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97269199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362800446548992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362800549536768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1607491331384832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97279190 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11227462761472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180564000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22577231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83284810 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40972611 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1596382609018880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0à*b‡42’"- _wandbruntime‚9006  _runtime‚9006§îç “"… trainer/global_step‚10499 3loss/FBt1/assignment_loss‚0.0021515782654798025 3loss/FBt2/assignment_loss‚0.0017843359292223954 1loss/SRqqt2/detection_loss‚0.5728794828433855 %loss/total_loss‚7.314279136493431 epoch‚26 % _timestamp‚1.7843493475381005e+09 _step‚865 /loss/FRt1/assignment_loss‚1.503804746809907 /loss/FBt2/detection_loss‚0.3599139739066189 _runtime‚9006.231397634 1loss/SRqqt2/assignment_loss‚0.521232290427124 1loss/SRqqt1/detection_loss‚0.7485841056331992 0loss/FRt2/assignment_loss‚0.6108786024939474 3loss/SRqqt1/assignment_loss‚0.48880081456553004 /loss/FRt1/detection_loss‚0.5526404023241384 .loss/FRt2/detection_loss‚1.388932567815198 /loss/FBt1/detection_loss‚0.5626762354796803wcJ3”"Š 1loss/SRqqt2/assignment_loss‚0.521232290427124 1loss/SRqqt1/detection_loss‚0.7485841056331992 0loss/FRt2/assignment_loss‚0.6108786024939474 3loss/SRqqt1/assignment_loss‚0.48880081456553004 /loss/FRt1/detection_loss‚0.5526404023241384 .loss/FRt2/detection_loss‚1.388932567815198 /loss/FBt1/detection_loss‚0.5626762354796803 trainer/global_step‚10499 3loss/FBt1/assignment_loss‚0.0021515782654798025 3loss/FBt2/assignment_loss‚0.0017843359292223954 1loss/SRqqt2/detection_loss‚0.5728794828433855 %loss/total_loss‚7.314279136493431 epoch‚26 % _timestamp‚1.7843493475381005e+09 _step‚865 /loss/FRt1/assignment_loss‚1.503804746809907 /loss/FBt2/detection_loss‚0.3599139739066189 _runtime‚9006.231397634á‡ÜÍËž•"‚(:“ ¬…ìÒäÈ“ gpu.0.gpu‚41.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dðÏé–"‚(:Þ ¬…ìÒŠ§ž cpu‚26.18238434970184 proc.cpu.threads‚17 disk.dm-0.in‚ 16137.8671875! disk.nvme0n1.out‚ 26885.203125 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 16137.8671875 disk.dm-0.out‚ 26885.203125 disk.sda2.in‚0! disk.nvme0n1.in‚ 16137.8671875 network.recv‚ 2.448898e+06 disk.rbd0.in‚0( proc.memory.availableMB‚ 478.60546875 disk.rbd1.out‚33.11767578125 disk.dm-1.out‚ 277.9921875* disk./.usagePercent‚39.488762016788854 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0$ disk./.usageGB‚588.4110908508301 disk.sda.in‚ 0.1328125 disk.sda.out‚ 277.9921875 disk.sda2.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2744.13671875 network.sent‚ 1.6762659e+07# disk.nvme0c0n1.out‚ 26885.203125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 277.9921875$ memory_percent‚97.07882404327393* proc.memory.percent‚16.748881340026855{¸ ‘—"‚(:† »…ìÒæÎÓ“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oä—ï˜"‚(:ä »…ìÒª˜î– network.sent‚ 1.6825841e+07 network.recv‚ 2.453885e+06' disk.nvme0c0n1.out‚26889.4736328125) disk./.usagePercent‚39.48873948812584$ disk./.usageGB‚588.4107551574707 disk.sda1.in‚0 disk.dm-1.out‚ 278.46875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 16137.8671875 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2744.16015625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 278.46875 disk.rbd1.out‚33.11767578125$ memory_percent‚97.08044528961182 disk.rbd0.in‚0 disk.dm-0.in‚ 16137.8671875" disk.dm-0.out‚26889.4736328125 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 278.46875* proc.memory.percent‚16.749024391174316 cpu‚26.182099824760776 disk.dm-1.in‚ 0.1328125 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 478.33984375 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0n1.in‚ 16137.8671875% disk.nvme0n1.out‚26889.4736328125 disk.rbd1.in‚ 2.01953125.ã’Í“™"‚(:ˆ Ê…ìÒÐᇘ gpu.0.gpu‚81.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀJbŽòš"‚(:ç Ê…ìÒ¥Þ‡ disk.dm-1.out‚ 278.73046875( disk.nvme0c0n1.out‚26895.91943359375 disk.drbd2165.in‚0# disk.dm-0.out‚26895.91943359375# memory_percent‚97.0804214477539! proc.memory.rssMB‚ 2744.171875 cpu‚26.181899132359213 network.recv‚ 2.457121e+06 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 16137.8671875 disk.sda2.in‚0! disk./.usageGB‚588.4111328125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 278.73046875* proc.memory.percent‚16.749095916748047 disk.dm-0.in‚ 16137.8671875 disk.sda.in‚ 0.1328125 disk.rbd1.out‚33.18017578125% proc.memory.availableMB‚ 478.34375 disk.sda2.out‚0! disk.nvme0n1.in‚ 16137.8671875& disk.nvme0n1.out‚26895.91943359375 disk.sda3.in‚ 0.1328125 network.sent‚ 1.6832971e+07) disk./.usagePercent‚39.48876483287172 disk.dm-1.in‚ 0.1328125 proc.cpu.threads‚17 disk.sda3.out‚ 278.73046875 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.sda1.out‚0˜á­l2›"- _wandbruntime‚9046  _runtime‚9046ã‚ÝÈœ" trainer/global_step‚10549 $ _timestamp‚1.784349387970768e+09 _runtime‚ 9046.66433991 _step‚866 ' lr-AdamW/pg1‚0.00021198443258690583 ' lr-AdamW/pg2‚0.00021198443258690583ãò™PÍ"Ç ' lr-AdamW/pg2‚0.00021198443258690583 trainer/global_step‚10549 $ _timestamp‚1.784349387970768e+09 _runtime‚ 9046.66433991 _step‚866 ' lr-AdamW/pg1‚0.00021198443258690583âx³2ž"- _wandbruntime‚9046  _runtime‚9046ëlQW"Ÿ"÷O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18481148 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364488750742528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1603597419470848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1603597436510208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364488860028928 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9046.665583308 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83658776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11158746579968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1603597419470848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83658778 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97714599 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41159890 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1614756183090176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1595348098592768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.s½‹-ͦmall_pool.allocated‚11141722784094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1603597436510208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22678742 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11277764085760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97714599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18481151 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353211095943168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614755824711680SX$p( "üO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364488860028928 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9046.665583308 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83658776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11158746579968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1603597419470848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83658778 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97714599 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41159890 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1614756183090176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1595348098592768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11141722784094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1603597436510208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22678742 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11277764085760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97714599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18481151 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353211095943168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614755824711680 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18481148 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364488750742528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1603597419470848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1603597436510208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438ã÷½Zh2¡"- _wandbruntime‚9047  _runtime‚9047L•¹×'¢"N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22680786 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353244524701696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83666408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1614904078368256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18482896 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1595494609264640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1595494626304000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843493885864804e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390059 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22681050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9047.280240662 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11159759214080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1603744660500480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1606637360141928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97733642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1603744677539840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11159417867776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1603744677539840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364523308254720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97723651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97733642 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11159759214080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83666408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41163679 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83666410 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11142733837928 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1614904436753920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18482893 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400052 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364523205266944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400052 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11159417867776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353244506089472 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97723651 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1603744660500480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1606637002552566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚10549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390059~¤,¯ '£"†N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83666410 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11142733837928 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1614904436753920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18482893 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400052 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364523205266944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400052 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11159417867776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353244506089472 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97723651 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1603744660500480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1606637002552566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚10549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390059 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22680786 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353244524701696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83666408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1614904078368256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18482896 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1595494609264640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1595494626304000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843493885864804e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390059 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22681050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9047.280240662 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11159759214080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1603744660500480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1606637360141928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97733642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1603744677539840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11159417867776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1603744677539840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364523308254720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97723651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97733642 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11159759214080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83666408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41163679 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088äk$ñ·2¤"- _wandbruntime‚9047  _runtime‚9047™a5¥"ˆ 0loss/FRt2/assignment_loss‚0.5844589031260946 /loss/FRt1/detection_loss‚0.5748043875535159 %loss/total_loss‚7.891756388684516 0loss/FRt1/assignment_loss‚1.5133863239772865 2loss/FBt2/assignment_loss‚0.004701356439462598 /loss/FRt2/detection_loss‚1.5585838588135665 1loss/SRqqt1/detection_loss‚0.7540532649693571 epoch‚26 % _timestamp‚1.7843493885890493e+09 _step‚869 3loss/FBt1/assignment_loss‚0.0007840538718618007 1loss/SRqqt2/detection_loss‚0.6262422914400936 /loss/FBt1/detection_loss‚0.8463052496488672 0loss/FBt2/detection_loss‚0.45151942759702884 trainer/global_step‚10549 _runtime‚9047.282344115 2loss/SRqqt1/assignment_loss‚0.5148302592690585 3loss/SRqqt2/assignment_loss‚0.46208701197832186F™‹š›¦" 2loss/FBt2/assignment_loss‚0.004701356439462598 /loss/FRt2/detection_loss‚1.5585838588135665 1loss/SRqqt1/detection_loss‚0.7540532649693®>|x571 epoch‚26 % _timestamp‚1.7843493885890493e+09 _step‚869 3loss/FBt1/assignment_loss‚0.0007840538718618007 1loss/SRqqt2/detection_loss‚0.6262422914400936 /loss/FBt1/detection_loss‚0.8463052496488672 0loss/FBt2/detection_loss‚0.45151942759702884 trainer/global_step‚10549 _runtime‚9047.282344115 2loss/SRqqt1/assignment_loss‚0.5148302592690585 3loss/SRqqt2/assignment_loss‚0.46208701197832186 0loss/FRt2/assignment_loss‚0.5844589031260946 /loss/FRt1/detection_loss‚0.5748043875535159 %loss/total_loss‚7.891756388684516 0loss/FRt1/assignment_loss‚1.5133863239772865åc ’§"‚(:‡ Ù…ìÒãŒô” gpu.0.gpu‚27.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ObñYí¨"‚(:â Ù…ìÒ“„Ì– disk.sda2.in‚0 disk.sda3.in‚ 0.1328125# disk.nvme0n1.out‚26899.01953125' proc.memory.availableMB‚ 478.3203125 network.recv‚ 2.46065e+06) disk./.usagePercent‚39.48877225709022$ disk./.usageGB‚588.4112434387207 disk.sda.out‚ 279.01953125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚33.18017578125! proc.memory.rssMB‚ 2744.171875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 16137.8671875 disk.sda3.out‚ 279.01953125 proc.cpu.threads‚17 network.sent‚ 1.6835935e+07 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 16137.8671875 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚26899.01953125 disk.dm-0.out‚26899.01953125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚97.08056449890137* proc.memory.percent‚16.749095916748047 cpu‚26.18163798874021 disk.dm-1.out‚ 279.01953125 disk.dm-0.in‚ 16137.8671875 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125þ)+B’©"‚(:‡ è…ìÒ‚›Ò“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚199.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.805 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09êù`úª"‚(:ï è…ìÒð–¨– disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 279.69921875 disk.rbd1.out‚ 33.3291015625 network.recv‚ 2.465132e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 279.69921875 disk.dm-0.in‚ 16199.6015625& disk.nvme0n1.out‚26904.24951171875 disk.rbd0.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚26904.24951171875 disk.sda.out‚ 279.69921875# proc.memory.rssMB‚ 2744.19140625 cpu‚26.181367780907824 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0$ disk./.usageGB‚588.4112548828125! disk.nvme0n1.in‚ 16199.6015625$ memory_percent‚97.08125591278076( proc.memory.availableMB‚ 478.20703125 disk.rbd1.in‚ 2.01953125) disk./.usagePercent‚39.48877302511282# disk.nvme0c0n1.in‚ 16199.6015625 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.749215126037598 network.sent‚ 1.6898973e+07 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚26904.24951171875 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17OÊ—2«"- _wandbruntime‚9087  _runtime‚9087ÁÄȬ" % _timestamp‚1.7843494290457382e+09 _runtime‚9087.739348118 _step‚870 & lr-AdamW/pg1‚0.0002101881585406756 & lr-AdamW/pg2‚0.0002101881585406756 trainer/global_step‚10599è;©Í­"Ç _runtime‚9087.739348118 _step‚870 & lr-AdamW/pg1‚0.0002101881585406756 & lr-AdamW/pg2‚0.0002101881585406756 trainer/global_step‚10599 % _timestamp‚1.7843494290457382e+09æÓSiH2®"- _wandbruntime‚9087  _runtime‚9087\~gü'¯"öO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22782503 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11329076011008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1613865988496730 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9087.740239321 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98169051 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366211685874688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354882533869568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚871 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182209427 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98179028 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18568401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1622168930073600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1602673632186368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1622168930073600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11329152005120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182219406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41350901 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98179028 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18568398 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84040376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1610959470952448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84040376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41351157 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1622168571695104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11191998721112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209442081792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366211591268352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354882515257344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1613865630907480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192339271002 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610959470952448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209442081792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚10599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1622168571695104 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1610959487991808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22782756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84040378 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610959487991808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888œÛ­(°"ûO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209442081792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366211591268352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354882515257344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1613865630907480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192339271002 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610959470952448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209442081792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚10599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1622168571695104 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1610959487991808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22782756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84040378 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610959487991808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22782503 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11329076011008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1613865988496730 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9087.740239321 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98169051 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366211685874688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354882533869568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚871 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182209427 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98179028 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18568401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1622168930073600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1602673632186368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1622168930073600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11329152005120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182219406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41350901 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98179028 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18568398 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84040376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1610959470952448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84040376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41351157 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1622168571695104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11191998721112çûN{‡2±"- _wandbruntime‚9088  _runtime‚9088[šÚú²"€N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41355007 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11193350324836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18570143 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11193009774834 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84048010 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1611106711982080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366246049100288 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1622317183737344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98178103 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366246147893760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210454715904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1622316825351680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98178103 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18570146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22784861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354915962628096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1611106729021440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1622317183737344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1602820159897600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11210113369600 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843494296623366e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1614013510222436 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1614013152633074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11330185265664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84048008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226111 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226111 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84048010 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1611106711982080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354915944015872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210454715904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98188094 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚9088.356077808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98188094 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11330105084416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41354746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888ª¦}T '³"…N WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41355007 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11193350324836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18570143 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11193009774834 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84048010 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1611106711982080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366246049100288 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1622317183737344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98178103 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366246147893760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210454715904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1622316825351680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98178103 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18570146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22784861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354915962628096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1611106729021440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1622317183737344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1602820159897600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11210113369600 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843494296623366e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1614013510222436 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1614013152633074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11330185265664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84048008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226111 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226111 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84048010 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1611106711982080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354915944015872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210454715904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98188094 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚9088.356077808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98188094 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11330105084416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41354746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0èg×ÐX2´"- _wandbruntime‚9088  _runtime‚9088¤¡Â µ"† _step‚873 0loss/FRt2/assignment_loss‚0.7096785030038549 2loss/SRqqt1/assignment_loss‚0.5354517423774855 2loss/FBt2/assignment_loss‚0.005576858224348894 1loss/SRqqt1/detection_loss‚0.7567232302019626 1loss/SRqqt2/detection_loss‚0.6018281753323622 4loss/FBt1/assignment_loss‚2.9142673161532328e-05 .loss/FBt1/detection_loss‚0.540267254319042 %loss/total_loss‚7.529770355836723 0loss/FRt1/assignment_loss‚1.5066680656391487 2loss/SRqqt2/assignment_loss‚0.4577624418175603 /loss/FRt1/detection_loss‚0.5714893274667388 trainer/global_step‚10599 % _timestamp‚1.7843494296663566e+09 _runtime‚9088.359637829 /loss/FRt2/detection_loss‚1.4579098898755765 /loss/FBt2/detection_loss‚0.3863857249054811 epoch‚262iÈ?¶"‹ %loss/total_loss‚7.529770355836723 0loss/FRt1/assignment_loss‚1.5066680656391487 2loss/SRqqt2/assignment_loss‚0.4577624418175603 /loss/FRt1/detection_loss‚0.5714893274667388 trainer/global_step‚10599 % _timestamp‚1.7843494296663566e+09 _runtime‚9088.359637829 /loss/FRt2/detection_loss‚1.4579098898755765 /loss/FBt2/detection_loss‚0.3863857249054811 epoch‚26 _step‚873 0loss/FRt2/assignment_loss‚0.7096785030038549 2loss/SRqqt1/assignment_loss‚0.5354517423774855 2loss/FBt2/assignment_loss‚0.005576858224348894 1loss/SRqqt1/detection_loss‚0.7567232302019626 1loss/SRqqt2/detection_loss‚0.6018281753323622 4loss/FBt1/assignment_loss‚2.9142673161532328e-05 .loss/FBt1/detection_loss‚0.540267254319042ét$“·"‚(:ˆ ÷…ìÒŠªå“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀUÃð¸"‚(:å ÷…ìÒª°Ò—$ disk./.usageGB‚588.4108238220215# proc.memory.rssMB‚ 2744.19140625* proc.memory.percent‚16.749215126037598 network.recv‚ 2.468641e+06 disk.dm-0.in‚ 16199.6015625 disk.sda.in‚ 0.1328125# disk.nvme0n1.out‚26907.65234375 disk.sda3.out‚ 279.92578125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 16199.6015625% disk.nvme0c0n1.out‚26907.65234375( proc.memory.availableMB‚ 478.00390625 network.sent‚ 1.6903946e+07! disk.nvme0n1.in‚ 16199.6015625 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.dm-0.out‚26907.65234375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 33.3291015625$ memory_percent‚97.08249568939209 proc.cpu.threads‚17) disk./.usagePercent‚39.48874409626145 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 279.92578125 disk.sda2.in‚0 cpu‚26.18097941074348 disk.sda1.in‚0 disk.dm-1.out‚ 279.92578125ᓜܹ"‚(:‘ ††ìÒјʕ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚160.173' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.034600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÙ>¼öº"‚(:ë ††ìÒÜáø–) disk./.usagePercent‚39.48874947241967 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.749215126037598$ disk./.usageGB‚588.4109039306641 disk.sda.out‚ 280.19921875$ memory_percent‚97.08008766174316' proc.memory.availableMB‚ 478.3984375 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0! disk.nvme0n1.in‚ 16199.6015625 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0" disk.dm-0.out‚26912.2685546875 disk.sda3.in‚ 0.1328125 network.recv‚ 2.472905e+06 disk.dm-1.out‚ 280.19921875# disk.nvme0c0n1.in‚ 16199.6015625 disk.dm-0.in‚ 16199.6015625 disk.sda3.out‚ 280.19921875 disk.rbd1.out‚ 33.3916015625 network.sent‚ 1.6968979e+07 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚26912.2685546875 disk.sda2.in‚0 cpu‚26.180768369596446% disk.nvme0n1.out‚26912.2685546875# proc.memory.rssMB‚ 2744.19140625 proc.cpu.threads‚17hRÛΑ»"‚(:† •†ìÒ®Ú€” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ßð9ò¼"‚(:ç •†ìÒ‰• – network.sent‚ 1.6971901e+07 disk.rbd1.out‚ 33.3916015625 disk.sda3.out‚ 280.57421875 disk.sda2.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.08166122436523& proc.memory.availableMB‚ 478.140625! disk.nvme0n1.in‚ 16199.6015625 disk.dm-1.out‚ 280.57421875# proc.memory.rssMB‚ 2744.20703125) proc.memory.percent‚16.74931049346924 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚26915.2529296875 network.recv‚ 2.476397e+06$ disk./.usageGB‚588.4108734130859" disk.dm-0.out‚26915.2529296875 disk.sda.out‚ 280.57421875% disk.nvme0n1.out‚26915.2529296875( disk./.usagePercent‚39.4887474243594 disk.dm-0.in‚ 16199.6015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 16199.6015625 cpu‚26.18047605467453 proc.cpu.threads‚17{\[2½"- _wandbruntime‚9128  _runtime‚9128}Ej‰È¾" % _timestamp‚1.7843494698606803e+09 _runtime‚ 9128.5564298 _step‚874 ' lr-AdamW/pg1‚0.00020839280618822006 ' lr-AdamW/pg2‚0.00020839280618822006 trainer/global_step‚10649QsWWÍ¿"Ç ' lr-AdamW/pg1‚0.00020839280618822006 ' lr-AdamW/pg2‚0.00020839280618822006 trainer/global_step‚10649 % _timestamp‚1.7843494698606803e+09 _runtime‚ 9128.5564298 _step‚874êo–ÍX2À"- _wandbruntime‚9128  _runtime‚9128°ü¡tý'Á"÷O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11380488120320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98623503 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11242955757910 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 % _timestamp‚1.7843494698637216e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367934459916288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22886597 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1609999182819328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9128.560424103 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84421978 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11259796244480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356553971795968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055458 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356553953183744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98623503 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1618321539473408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11259796244480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1618321539473408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41542245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629581677057024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11260137583616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1618321522434048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚875 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11242615208020 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045479 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629581318678528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1621241780987988 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41542517 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11380403737600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1621242138577238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11260137583616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629581318678528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1618321522434048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84421976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18655651 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367934356921344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84421978 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11259796244480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356553971795968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055458 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356553953183744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98623503 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1618321539473408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11259796244480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1618321539473408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41542245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629581677057024 \IDeviceStatsMonitor.on²©»ÿm_train_batch_start/active_bytes.small_pool.allocated‚11260137583616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1618321522434048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚875 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11242615208020 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045479 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629581318678528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1621241780987988 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41542517 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11380403737600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1621242138577238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11260137583616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629581318678528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1618321522434048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84421976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18655651 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367934356921344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98623503 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11242955757910 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 % _timestamp‚1.7843494698637216e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367934459916288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22886597 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1609999182819328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9128.560424103 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ëËê42Ã"- _wandbruntime‚9129  _runtime‚9129šê+'Ä"ÿM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41546320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629729572335104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610145676451840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11243966811744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1621389660302944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1618468780503040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41546063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629729930720768 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356587381942272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98642546 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1618468780503040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1621389302713582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98632555 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11260808871424 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1618468763463680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183062163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11381423699456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚876 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11261150217728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 9129.17609207 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629729930720768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367968805641728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1618468763463680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367968910726656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11381510172160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84429610 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11260808871424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183062163 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84429608 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784349470482289e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072156 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18657396 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088ç‘Ø 'Å"„N EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚876 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11261150217728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 9129.17609207 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629729930720768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367968805641728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1618468763463680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367968910726656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11381510172160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84429610 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11260808871424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183062163 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84429608 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784349470482289e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072156 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18657396 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41546320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629729572335104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610145676451840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11243966811744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1621389660302944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1618468780503040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41546063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629729930720768 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356587381942272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98642546 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1618468780503040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1621389302713582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98632555 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11260808871424 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1618468763463680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183062163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11381423699456ìWsŸß2Æ"- _wandbruntime‚9129  _runtime‚9129}d´ß Ç"„ /loss/FRt2/detection_loss‚1.5395458282937762 1loss/SRqqt1/detection_loss‚0.7971075418509308 /loss/FBt1/detection_loss‚0.6337360216191772 % _timestamp‚1.7843494704848046e+09 3loss/FBt2/assignment_loss‚0.0003465700344434782 trainer/global_step‚10649 2loss/FBt1/assignment_loss‚0.004333292627443947 _runtime‚9129.178122124 _step‚877 2loss/SRqqt1/assignment_loss‚0.5064200927298149 1loss/SRqqt2/assignment_loss‚0.480604549377859 /loss/FRt1/detection_loss‚0.5660416675540773 1loss/SRqqt2/detection_loss‚0.6404389302227395 /loss/FBt2/detection_loss‚0.4134297521377448 %loss/total_loss‚7.703642818436557 epoch‚26 /loss/FRt1/assignment_loss‚1.525346751134601 0loss/FRt2/assignment_loss‚0.5962918208539486,,dÈ"‰ 0loss/FRt2/assignment_loss‚0.5962918208539486 /loss/FRt2/detection_loss‚1.5395458282937762 1loss/SRqqt1/detection_loss‚0.7971075418509308 /loss/FBt1/detection_loss‚0.6337360216191772 % _timestamp‚1.7843494704848046e+09 3loss/FBt2/assignment_loss‚0.0003465700344434782 trainer/global_step‚10649 2loss/FBt1/assignment_loss‚0.004333292627443947 _runtime‚9129.178122124 _step‚877 2loss/SRqqt1/assignment_loss‚0.5064200927298149 1loss/SRqqt2/assignment_loss‚0.480604549377859 /loss/FRt1/detection_loss‚0.5660416675540773 1loss/SRqqt2/detection_loss‚0.6404389302227395 /loss/FBt2/detection_loss‚0.4134297521377448 %loss/total_loss‚7.703642818436557 epoch‚26 /loss/FRt1/assignment_loss‚1.525346751134601í) 'À“É"‚(:ˆ ¤†ìÒ¦ÿµ• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚191.858' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.3716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yq`´õÊ"‚(:ê ¤†ìÒØ‘•– disk.sda1.out‚0 disk.rbd1.out‚ 33.4541015625 network.recv‚ 2.479932e+06 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚26919.50244140625 disk.sda.in‚ 0.1328125 disk.sda.out‚ 280.8359375 disk.sda2.out‚0 network.sent‚ 1.6976938e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 280.8359375 disk.dm-0.in‚ 16269.9765625 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 16269.9765625( proc.memory.availableMB‚ 477.62890625 disk.drbd2165.in‚0 disk.sda1.in‚0# disk.dm-0.out‚26919.50244140625! disk.nvme0n1.in‚ 16269.9765625$ memory_percent‚97.08478450775146 disk.sda3.out‚ 280.8359375 disk.rbd1.in‚ 2.01953125 cpu‚26.180126731803043 disk.drbd2165.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚26919.50244140625) proc.memory.percent‚16.74933433532715" proc.memory.rssMB‚ 2744.2109375) disk./.usagePercent‚39.48874179219365$ disk./.usageGB‚588.4107894897461‡ÃW‹žË"‚(:“ ³†ìÒÞÀÀ“ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.713' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.942600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹=¯ñÌ"‚(:æ ³†ìÒÀÎÓ•" proc.memory.rssMB‚ 2744.2109375* disk./.usagePercent‚39.488742816223784 disk.sda1.out‚0 cpu‚26.179726386018615 network.recv‚ 2.484393e+06 disk.sda1.in‚0 disk.dm-0.in‚ 16269.9765625 disk.sda.in‚ 0.1328125& proc.memory.availableMB‚ 477.671875 disk.drbd2165.in‚0 disk.sda.out‚ 281.08984375$ memory_percent‚97.08452224731445$ disk./.usageGB‚588.4108047485352 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 disk.dm-1.out‚ 281.08984375& disk.nvme0c0n1.out‚26922.173828125 disk.sda2.in‚0$ disk.nvme0n1.out‚26922.173828125 disk.rbd1.out‚ 33.4541015625) proc.memory.percent‚16.74933433532715 network.sent‚ 1.7039861e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 16269.9765625! disk.dm-0.out‚26922.173828125! disk.nvme0n1.in‚ 16269.9765625 disk.sda3.out‚ 281.08984375ÙΓÍ"‚(:ˆ †ìÒ±ºã“ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.382' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥š•ôÎ"‚(:é †ìÒ„¢ü•$ disk./.usageGB‚588.4072456359863 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.dm-0.out‚26927.75927734375 disk.rbd1.out‚ 33.5166015625# memory_percent‚97.0832109451294 network.recv‚ 2.487705e+06 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 16269.9765625 disk.sda.in‚ 0.1328125 cpu‚26.179322425169694) disk./.usagePercent‚39.48850396119434 disk.rbd0.out‚0( disk.nvme0c0n1.out‚26927.75927734375! disk.nvme0n1.in‚ 16269.9765625 disk.sda3.in‚ 0.1328125) proc.memory.percent‚16.74933433532715 network.sent‚ 1.7044985e+07 disk.rbd0.in‚0 disk.dm-1.out‚ 281.3671875 disk.drbd2165.out‚0 disk.sda3.out‚ 281.3671875 disk.sda.out‚ 281.3671875 disk.sda2.in‚0& disk.nvme0n1.out‚26927.75927734375 disk.rbd1.in‚ 2.01953125( proc.memory.availableMB‚ 477.88671875" proc.memory.rssMB‚ 2744.2109375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 16269.9765625 disk.sda2.out‚0 proc.cpu.threads‚17åðµ‡2Ï"- _wandbruntime‚9169  _runtime‚9169h,¾–ÆÐ"À & lr-AdamW/pg1‚0.0002065984908416658 & lr-AdamW/pg2‚0.0002065984908416658 trainer/global_step‚10699 # _timestamp‚1.78434951083675e+09 _runtime‚9169.535464361 _step‚878SòËÑ"Å & lr-AdamW/pg1‚0.0002065984908416658 & lr-AdamW/pg2‚0.0002065984908416658 trainer/global_step‚10699 # _timestamp‚1.78434951083675e+09 _runtime‚9169.535464361 _step‚878îØFa2Ò"- _wandbruntime‚9169  _runtime‚9169Gw²döÓ"÷O S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1636994065661952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pÄù¸–'ool.freed‚11431761311232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628618288657746 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9169.536425867 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628617931068496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1617324699373568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41733107 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18742901 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1625683573915648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1625683573915648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369657259610624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41732829 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84803576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077955 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1625683590955008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11310833085440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚879 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11310833085440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183891510 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84803578 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369657152421376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881531 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1636994065661952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11431849888256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11293572244818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22989931 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84803576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84803578 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99087932 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1625683590955008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18742898 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077955 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496œÛ´(Ô"üO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881531 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1636994065661952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11431849888256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11293572244818 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22989931 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84803576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84803578 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99087932 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1625683590955008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18742898 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077955 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1636994065661952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11431761311232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628618288657746 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9169.536425867 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628617931068496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1617324699373568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41733107 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18742901 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1625683573915648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1625683573915648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369657259610624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41732829 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84803576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077955 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1625683590955008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11310833085440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚879 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11310833085440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183891510 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84803578 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369657152421376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438ïnþ|2Õ"- _wandbruntime‚9170  _runtime‚9170¦t¬Å'Ö"€N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99096998 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41736559 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369691618873344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11311845719552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358258838480896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41736818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11311504373248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1625830814945280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11432799004672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1637142677704192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18744646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1637142319318528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1628765810383452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1628765452794090 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369691713472512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11311504373248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99096998 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚9170.150313074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99087007 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚10699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84811210 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11311845719552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294242748650 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11432874991616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Kg›œ ×"…N BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41736559 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pooµUÈPol.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369691618873344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11311845719552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358258838480896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41736818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11311504373248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1625830814945280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11432799004672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1637142677704192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18744646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1637142319318528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1628765810383452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1628765452794090 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369691713472512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11311504373248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99096998 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚9170.150313074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99087007 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚10699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84811210 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11311845719552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294242748650 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11432874991616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99096998 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717583 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11361187248128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1633045642436608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1624650232967168 trainer/global_step‚10749 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99542384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚883 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41923682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18830148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41923941 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18830151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23093790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11361187248128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371380104165376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371380007461888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11344188731726 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843495516595352e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1644406812645376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1633045625397248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11361528587264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1635994438738254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1635994081149004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11343848181836 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1633045625397248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85185178 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1644407171023872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359896847648768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717583 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11361187248128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1633045642436608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1624650232967168 trainer/global_step‚10749 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99542384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚883 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41923682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18830148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41923941 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18830151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23093790 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11361187248128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371380104165376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371380007461888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11344188731726 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843495516595352e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1644406812645376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1633045625397248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11361528587264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1635994438738254 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1635994081149004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11343848181836 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1633045625397248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85185178 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1644407171023872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359896847648768 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371414563793920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359930276407296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85192808 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1636141960463960 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1636141602874598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99551450 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11362541221376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11345199785560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1644555066301952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1633192866426880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85192808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23095587 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18831893 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1624796760678400 _runtime‚9210.969517462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11362199875072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85192810 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41927480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1644555424687616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184734267 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359930257795072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚10749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11362199875072 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23095843 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976‘!§ 'é"…N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371414467097600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843495522755494e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184744260 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11344859235558 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371414563793920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359930276407296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85192808 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1636141960463960 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1636141602874598 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99551450 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11362541221376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11345199785560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1644555066301952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1633192866426880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85192808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23095587 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18831893 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1624796760678400 _runtime‚9210.969517462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11362199875072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85192810 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41927480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1644555424687616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184734267 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359930257795072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚10749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11362199875072 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23095843 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99986859 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1640407676878848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1631975783600128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10799 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11394464668744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11412224089088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42114586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStabnˆitsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11534477474304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1640407693918208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11411882749952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18917401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361568266962944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99996836 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11411882749952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373102857917952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18917398 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1631975766560768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843495926425922e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11534572342784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1651819559628800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42114306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23196908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 _step‚887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983‹ÎŽ(ö"üO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1651819559628800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42114306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23196908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 _step‚887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185563614 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85566776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85566778 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1640407676878848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85566778 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99986859 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1640407676878848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1631975783600128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10799 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11394464668744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11412224089088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42114586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11534477474304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1640407693918208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11411882749952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18917401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361568266962944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99996836 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11411882749952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373102857917952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18917398 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1631975766560768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843495926425922e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11534572342784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1÷„cî¬2÷"- _wandbruntime‚9251  _runtime‚9251wé‚ 'ø"ƒN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1651968171671040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99995911 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373137193706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85574408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42118354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18919146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23199208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843495932610471e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23198920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚10799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11412895376896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18919143 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373137302985728 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1640554934947840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643517752955106 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11395475722466 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185570319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85574410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85574410 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42118063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361601714333696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1632122294272000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11535497985024 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643518110544468 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚9251.954847036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570319 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1640554934947840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1651967813285376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1640554917908480 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11413236723200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11412895376896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100005902 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643517752955106 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11395475722466 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185570319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85574410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85574410 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42118063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361601714333696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1632122294272000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11535497985024 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643518110544468 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚9251.954847036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570319 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1640554934947840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1651967813285376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1640554917908480 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11413236723200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11412895376896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100005902 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99995911 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_py”cþool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373137193706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85574408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42118354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18919146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23199208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843495932610471e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23198920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚10799ø—¢Ò2ú"- _wandbruntime‚9251  _runtime‚9251I ý û"… 3loss/SRqqt2/assignment_loss‚0.47551213711968016 /loss/FRt1/detection_loss‚0.5925090959256814 1loss/SRqqt2/detection_loss‚0.6763815046371625 /loss/FBt1/detection_loss‚0.4789461779793532 _runtime‚9251.956975204 4loss/FBt1/assignment_loss‚0.00013294821868078285 3loss/FBt2/assignment_loss‚0.0040484032772171155 /loss/FRt2/detection_loss‚1.4421133117805722 1loss/SRqqt1/detection_loss‚0.7719346969563892 /loss/FBt2/detection_loss‚0.4265715393748553 %loss/total_loss‚7.528959017045737 epoch‚26 trainer/global_step‚10799 # _timestamp‚1.78434959326363e+09 2loss/SRqqt1/assignment_loss‚0.5090183065291268 _step‚889 .loss/FRt1/assignment_loss‚1.52055177749408 0loss/FRt2/assignment_loss‚0.6312391177529381ªÎ´ü"Š /loss/FBt1/detection_loss‚0.4789461779793532 _runtime‚9251.956975204 4loss/FBt1/assignment_loss‚0.00013294821868078285 3loss/FBt2/assignment_loss‚0.0040484032772171155 /loss/FRt2/detection_loss‚1.4421133117805722 1loss/SRqqt1/detection_loss‚0.7719346969563892 /loss/FBt2/detection_loss‚0.4265715393748553 %loss/total_loss‚7.528959017045737 epoch‚26 trainer/global_step‚10799 # _timestamp‚1.78434959326363e+09 2loss/SRqqt1/assignment_loss‚0.5090183065291268 _step‚889 .loss/FRt1/assignment_loss‚1.52055177749408 0loss/FRt2/assignment_loss‚0.6312391177529381 3loss/SRqqt2/assignment_loss‚0.47551213711968016 /loss/FRt1/detection_loss‚0.5925090959256814 1loss/SRqqt2/detection_loss‚0.6763815046371625ù–JŒœý"‚(:‘ œ‡ìÒîÇÉ“ gpu.0.gpu‚20.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.047' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.60940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý©¡óþ"‚(:è œ‡ìÒˆÖº— network.sent‚ 1.7190923e+07 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 283.859375& disk.nvme0n1.out‚26954.36962890625( proc.memory.availableMB‚ 474.56640625$ disk./.usageGB‚588.4109764099121 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚26954.36962890625" disk.nvme0n1.in‚16378.69140625 disk.sda3.out‚ 283.859375 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚16378.69140625( disk.nvme0c0n1.out‚26954.36962890625 disk.sda.out‚ 283.859375$ memory_percent‚97.10347652435303 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0! proc.memory.rssMB‚ 2746.765625 cpu‚26.17884098899603) disk./.usagePercent‚39.48875433656283 disk.sda1.out‚0 disk.rbd1.out‚ 33.7666015625 network.recv‚ 2.509997e+06 disk.sda3.in‚ 0.1328125) proc.memory.percent‚16.76492691040039 disk.dm-0.in‚16378.69140625 disk.rbd1.in‚ 2.01953125€Ãˆÿ"‚(:„ «‡ìқ˖• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.16' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÛV†ù€#‚(:î «‡ìÒáÿ– disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 284.18359375 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 33.7666015625 disk.dm-1.out‚ 284.18359375 disk.sda.out‚ 284.18359375 disk.dm-1.in‚ 0.1328125) proc.memory.percent‚16.77250862121582 network.sent‚ 1.7255238e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚16378.69140625) disk./.usagePercent‚39.48876457686419 disk.dm-0.in‚16378.69140625& disk.nvme0n1.out‚26957.41259765625 disk.sda1.in‚0 cpu‚26.183076977235416" disk.nvme0n1.in‚16378.69140625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 2.51401e+06$ disk./.usageGB‚588.4111289978027 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26957.41259765625' proc.memory.availableMB‚ 474.3359375# disk.dm-0.out‚26957.41259765625$ memory_percent‚97.10488319396973" proc.memory.rssMB‚ 2748.0078125 proc.cpu.threads‚17n"½2#- _wandbruntime‚9270  _runtime‚9270§Þ“Æw=‚#ñz 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.646845281124115 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33755481243133545 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07396776229143143 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7166795134544373 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7730464935302734 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6618221998214722 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02126242220401764 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/sensitivity‚0.6168333888053894 7validation_average_jet_accuracy‚0.38077297806739807 _step‚890 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8362622857093811 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08231977373361588 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_3_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7381572127342224 +jet/accuracy_2_of_2‚0.14176400005817413 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7508558630943298 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5873316526412964 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21698647737503052 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5004222393035889 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955416917800903 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29838746786117554 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5450944900512695 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2394675314426422 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6950050592422485 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6536901593208313 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7562090754508972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039303939789533615 *jet/accuracy_1_of_2‚0.7749271392822266 /particle/accuracy_4_of_4‚0.9580487608909607 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.778444766998291 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22073659300804138 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_1‚0.3466164767742157 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3151281774044037 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6794954538345337 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02397002838551998 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035760506987571716 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2783730924129486 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6977881193161011 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 epoch‚26 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972747564315796 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7055972218513489 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3065718412399292 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30393892526626587 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11918072402477264 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6208876371383667 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6583165526390076 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9080929756164551 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975546598434448 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.240213081240654 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7313669919967651 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8299390077590942 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5794727206230164 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29888632893562317 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.374982088804245 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33459773659706116 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02494145557284355 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7571728825569153 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6350308656692505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚9270.578485644 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.622940182685852 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07546810060739517 particle/accuracy_1_of_3‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21610558032989502 particle/accuracy_2_of_4‚1 trainer/global_step‚10799 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02419382892549038 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8998080492019653 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.94476717710495 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4319930672645569 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04291979596018791 (particle/accuracy‚0.8390377759933472 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975546598434448 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2874852418899536 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7868261933326721 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8422147035598755 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28507280349731445 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980757832527161 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2191353589296341 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7361352443695068 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6870700120925903 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.99500972032547 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0589170902967453 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3447861075401306 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8299319744110107 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7848712205886841 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2873312830924988 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8041501641273499 *jet/accuracy_1_of_4‚0.9831177592277527 'particle/f_score‚0.6220399141311646 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965059161186218 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965319633483887 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22827798128128052 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10015285760164261 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8820316791534424 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8818562626838684 $ _timestamp‚1.784349611884102e+09 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6466943025588989 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4999559819698334 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024638989940285683 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.803768515586853 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8274187445640564 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_4_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8521542549133301 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2088928520679474 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6961809396743774 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7372806668281555 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02126242220401764 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8405532836914062 )loss/val_total_loss‚7.636138984171514 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990808367729187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6358641386032104 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29237911105155945 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5016182065010071 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3847392499446869 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09012730419635773 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6476964354515076 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7462605834007263 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48734819889068604 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7595049738883972 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10122910887002945 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37231528759002686 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.673811674118042 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962862730026245 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2232983261346817 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6350308656692505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22702684998512268 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6786625385284424 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2874852418899536 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2068699598312378 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6719647645950317 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2336757928133011 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958664178848267 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8818562626838684 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7469637989997864 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04693160951137543 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.225723996758461 jet/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368«ºr”¨)ƒ#öz :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4319930672645569 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04291979596018791 (particle/accuracy‚0.8390377759933472 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975546598434448 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2874852418899536 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7868261933326721 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8422147035598755 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28507280349731445 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980757832527161 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2191353589296341 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7361352443695068 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6870700120925903 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.99500972032547 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0589170902967453 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3447861075401306 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8299319744110107 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7848712205886841 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2873312830924988 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8041501641273499 *jet/accuracy_1_of_4‚0.9831177592277527 'particle/f_score‚0.6220399141311646 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965059161186218 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965319633483887 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22827798128128052 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10015285760164261 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8820316791534424 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8818562626838684 $ _timestamp‚1.784349611884102e+09 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6466943025588989 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4999559819698334 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024638989940285683 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.803768515586853 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8274187445640564 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_4_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8521542549133301 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2088928520679474 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6961809396743774 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7372806668281555 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02126242220401764 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8405532836914062 )loss/val_total_loss‚7.636138984171514 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990808367729187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6358641386032104 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29237911105155945 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5016182065010071 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3847392499446869 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09012730419635773 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6476964354515076 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7462605834007263 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48734819889068604 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7595049738883972 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10122910887002945 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37231528759002686 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.673811674118042 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962862730026245 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2232983261346817 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6350308656692505 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22702684998512268 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6786625385284424 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2874852418899536 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2068699598312378 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6719647645950317 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2336757928133011 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958664178848267 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8818562626838684 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7469637989997864 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04693160951137543 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.225723996758461 jet/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.646845281124115 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33755481243133545 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.07396776229143143 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7166795134544373 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7730464935302734 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6618221998214722 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02126242220401764 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +particle/sensitivity‚0.6168333888053894 7validation_average_jet_accuracy‚0.38077297806739807 _step‚890 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8362622857093811 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08231977373361588 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_3_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7381572127342224 +jet/accuracy_2_of_2‚0.14176400005817413 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7508558630943298 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5873316526412964 8 Purity/2FåÀÔRt*SRqqt0FBt/FRt_purity‚0.21698647737503052 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5004222393035889 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955416917800903 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29838746786117554 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5450944900512695 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2394675314426422 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6950050592422485 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6536901593208313 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7562090754508972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039303939789533615 *jet/accuracy_1_of_2‚0.7749271392822266 /particle/accuracy_4_of_4‚0.9580487608909607 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.778444766998291 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22073659300804138 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_1‚0.3466164767742157 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3151281774044037 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6794954538345337 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02397002838551998 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035760506987571716 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2783730924129486 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6977881193161011 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 epoch‚26 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972747564315796 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7055972218513489 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3065718412399292 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30393892526626587 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11918072402477264 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6208876371383667 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6583165526390076 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9080929756164551 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975546598434448 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.240213081240654 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7313669919967651 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8299390077590942 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5794727206230164 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29888632893562317 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.374982088804245 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33459773659706116 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02494145557284355 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7571728825569153 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6350308656692505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚9270.578485644 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.622940182685852 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07546810060739517 particle/accuracy_1_of_3‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21610558032989502 particle/accuracy_2_of_4‚1 trainer/global_step‚10799 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02419382892549038 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8998080492019653 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.94476717710495úäÄ„#‚(:’ º‡ìÒÍ” gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÌ3õö…#‚(:ë º‡ìÒ¶íÑ• disk.dm-1.out‚ 284.3671875 disk.rbd0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 474.30859375" disk.nvme0n1.in‚16378.69140625& disk.nvme0n1.out‚26963.40869140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 284.3671875) proc.memory.percent‚16.77250862121582 cpu‚26.18308025922182 network.sent‚ 1.7258261e+07( disk.nvme0c0n1.out‚26963.40869140625# disk.dm-0.out‚26963.40869140625" proc.memory.rssMB‚ 2748.0078125 network.recv‚ 2.517296e+06 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda.in‚ 0.1328125) disk./.usagePercent‚39.48873359995255 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125# memory_percent‚97.1050500869751$ disk./.usageGB‚588.4106674194336 disk.sda3.out‚ 284.3671875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚16378.69140625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 33.8291015625 disk.dm-0.in‚16378.69140625 proc.cpu.threads‚17Æ*2꓆#‚(:ˆ ɇìÒü†ã“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚239.453' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8906 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â·\`ú‡#‚(:ï ɇìÒþá•– disk.rbd1.out‚ 33.8291015625 disk.drbd2165.out‚0$ memory_percent‚97.10488319396973 disk.sda.out‚ 284.76171875& disk.nvme0n1.out‚26966.49853515625 disk.dm-0.in‚16378.69140625' proc.memory.availableMB‚ 474.3359375 network.recv‚ 2.522066e+06 cpu‚26.182702379125192" disk.nvme0n1.in‚16378.69140625 disk.sda3.out‚ 284.76171875 disk.sda1.in‚0 disk.dm-1.out‚ 284.76171875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚26966.49853515625 network.sent‚ 1.7315661e+07 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.dm-0.out‚26966.49853515625$ disk./.usageGB‚588.4107856750488 disk.sda1.out‚0$ disk.nvme0c0n1.in‚16378.69140625) proc.memory.percent‚16.77250862121582) disk./.usagePercent‚39.48874153618611 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.sda2.out‚0õp72ˆ#- _wandbruntime‚9311  _runtime‚9311`osNʉ#Ä ' lr-AdamW/pg1‚0.00020122291891519682 ' lr-AdamW/pg2‚0.00020122291891519682 trainer/global_step‚10849 % _timestamp‚1.7843496523807766e+09 _runtime‚9311.074370759 _step‚891†Xº ÏŠ#É _step‚891 ' lr-AdamW/pg1‚0.00020122291891519682 ' lr-AdamW/pg2‚0.00020122291891519682 trainer/global_step‚10849 % _timestamp‚1.7843496523807766e+09 _runtime‚9311.074370759ûp‚é}2‹#- _wandbruntime‚9311  _runtime‚9311=c;ý(Œ#úO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363255635601408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86069042 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1649699295010816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1652672764249329 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11463141430771 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661179606468096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42338041 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100521610 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363255616989184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86069042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11480652796416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1649699312050176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100521610 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374859380288000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23330602 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1649699312050176 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11480311457280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590654 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100511633 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661179964846592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100511633 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9311.075355257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23330870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11480652796416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11480311457280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661179606468096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1649699295010816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1652672764249329 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11463141430771 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661179606468096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42338041 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100521610 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363255616989184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86069042 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11480652796416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1649699312050176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100521610 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374859380288000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23330602 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1649699312050176 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11480311457280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590654 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100511633 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661179964846592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100511633 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9311.075355257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23330870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11480652796416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11480311457280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661179606468096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23332976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363289045747712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42341892 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚893 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1661327860124672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1649846553079808 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11481324084224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100530676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚9311.692819547 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86076674 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374893830069248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363289064359936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843496529989376e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597359 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86076674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1641356491079680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1649846536040448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1661328218510336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186607352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1649846553079808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11481665430528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23332720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42341633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19008913 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088cãü'#ŠN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1649846553079808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11481665430528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23332720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42341633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19008913 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86076676 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1649846536040448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186597359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19008916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374893935154176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23332976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363289045747712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42341892 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚893 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1661327860124672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1649846553079808 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11481324084224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100530676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚9311.692819547 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86076674 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374893830069248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363289064359936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843496529989376e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597359 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86076674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1641356491079680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1649846536040448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1661328218510336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186607352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269ýtkp2‘#- _wandbruntime‚9311  _runtime‚9311o/´† ’#† 2loss/SRqqt1/assignment_loss‚0.5181592604389886 2loss/SRqqt2/assignment_loss‚0.4868978389584246 1loss/SRqqt1/detection_loss‚0.8584711306043468 1loss/SRqqt2/detection_loss‚0.5904076321086265 /loss/FBt1/detection_loss‚0.4867722306710978 trainer/global_step‚10849 _step‚894 3loss/FBt1/assignment_loss‚2.115952147521385e-06 /loss/FBt2/detection_loss‚0.5168682268695869 %loss/total_loss‚7.583165851952911 0loss/FRt2/assignment_loss‚0.5862338250433957 /loss/FRt1/detection_loss‚0.5462174218897712 % _timestamp‚1.7843496530020072e+09 0loss/FRt1/assignment_loss‚1.5314894590663726 2loss/FBt2/assignment_loss‚0.016145283149631023 /loss/FRt2/detection_loss‚1.4455014272005227 epoch‚27 _runtime‚9311.695326197u2b†“#‹ /loss/FBt1/detection_loss‚0.4867722306710978 trainer/global_step‚10849 _step‚894 3loss/FBt1/assignment_loss‚2.115952147521385e-06 /loss/FBt2/detection_loss‚0.5168682268695869 %loss/total_loss‚7.583165851952911 0loss/FRt2/assignment_loss‚0.5862338250433957 /loss/FRt1/detection_loss‚0.5462174218897712 % _timestamp‚1.7843496530020072e+09 0loss/FRt1/assignment_loss‚1.5314894590663726 2loss/FBt2/assignment_loss‚0.016145283149631023 /loss/FRt2/detection_loss‚1.4455014272005227 epoch‚27 _runtime‚9311.695326197 2loss/SRqqt1/assignment_loss‚0.5181592604389886 2loss/SRqqt2/assignment_loss‚0.4868978389584246 1loss/SRqqt1/detection_loss‚0.8584711306043468 1loss/SRqqt2/detection_loss‚0.5904076321086265þC™Ý“”#‚(:ˆ ؇ìÒú¡” gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™)p ü•#‚(:ñ ؇ìÒâŠÕ–$ disk./.usageGB‚588.4108123779297 disk.drbd2165.out‚0 disk.dm-0.in‚16433.95703125 disk.sda.in‚ 0.1328125 proc.cpu.threads‚17 network.sent‚ 1.7320699e+07 disk.rbd0.out‚0# disk.dm-0.out‚26972.92822265625 disk.sda.out‚ 285.15234375& disk.nvme0n1.out‚26972.92822265625 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚16433.95703125 disk.sda2.in‚0 disk.sda3.out‚ 285.15234375 disk.rbd1.out‚ 33.8916015625) proc.memory.percent‚16.77250862121582* disk./.usagePercent‚39.488743328238854 disk.sda1.out‚0( disk.nvme0c0n1.out‚26972.92822265625 disk.sda2.out‚0 cpu‚26.182272424348803 network.recv‚ 2.525461e+06 disk.sda1.in‚0" disk.nvme0n1.in‚16433.95703125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.10633754730225( proc.memory.availableMB‚ 474.09765625 disk.rbd0.in‚0 disk.dm-1.out‚ 285.15234375 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125ú~FÒœ–#‚(:‘ ç‡ìÒ¤àë“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.437' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.687399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¿ ù—#‚(:î ç‡ìÒï¤Ï• disk.sda2.out‚0 network.recv‚ 2.530035e+06) disk./.usagePercent‚39.48874512029159# disk.dm-0.out‚26975.77490234375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 285.37890625 disk.sda3.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125 cpu‚26.181986366096606$ disk.nvme0c0n1.in‚16433.95703125& disk.nvme0n1.out‚26975.77490234375 disk.sda3.out‚ 285.37890625 disk.rbd1.out‚ 33.8916015625" disk.nvme0n1.in‚16433.95703125( proc.memory.availableMB‚ 474.34765625 disk.sda1.out‚0( disk.nvme0c0n1.out‚26975.77490234375 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 285.37890625 proc.cpu.threads‚17 network.sent‚ 1.7383814e+07$ disk./.usageGB‚588.4108390808105 disk.rbd0.in‚0 disk.dm-0.in‚16433.95703125 disk.sda.in‚ 0.1328125" memory_percent‚97.104811668396) proc.memory.percent‚16.77250862121582 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0V" ê˜#‚(:’ ö‡ìÒ±¨„” gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚277.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»»;÷™#‚(:ì ö‡ìÒÐê«– disk.dm-1.out‚ 285.7421875 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚16433.95703125 disk.sda3.in‚ 0.1328125# memory_percent‚97.1036434173584' proc.memory.availableMB‚ 474.5390625 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda3.out‚ 285.7421875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚16433.95703125( disk.nvme0c0n1.out‚26980.08056640625& disk.nvme0n1.out‚26980.08056640625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2748.0078125 network.sent‚ 1.7388696e+07$ disk./.usageGB‚588.4107475280762* disk./.usagePercent‚39.488738976110774 disk.dm-0.in‚16433.95703125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 33.9541015625) proc.memory.percent‚16.77250862121582 disk.rbd0.in‚0 cpu‚26.181770978060484 network.recv‚ 2.533262e+06 disk.sda1.out‚0# disk.dm-0.out‚26980.08056640625 disk.sda.out‚ 285.7421875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17Ѭ;*2š#- _wandbruntime‚9352  _runtime‚9352Ù•¯Ê›#Ä ' lr-AdamW/pg1‚0.00019943390327041503 ' lr-AdamW/pg2‚0.00019943390327041503 trainer/global_step‚10899 % _timestamp‚1.7843496933967075e+09 _runtime‚9352.090244994 _step‚895ø`Þ#Ïœ#É trainer/global_step‚10899 % _timestamp‚1.7843496933967075e+09 _runtime‚9352.090244994 _step‚895 ' lr-AdamW/pg1‚0.00019943390327041503 ' lr-AdamW/pg2‚0.00019943390327041503ÿ*mQ02#- _wandbruntime‚9352  _runtime‚9352 ocî&ž#úO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1657061363531776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1668592711830016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1648535514001408 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1668592353451520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531006959104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531006959104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19094418 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11531348298240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86450644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23434611 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376582157540352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11655102624768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11513757917679 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100976062 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1657061363531776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.cuÊ’FŒrrent‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11531348298240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Xu”(Ÿ#ÿO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1657061363531776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1668592711830016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1648535514001408 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1668592353451520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531006959104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531006959104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19094418 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11531348298240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86450644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23434611 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376582157540352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11655102624768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11513757917679 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100976062 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1657061363531776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11531348298240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977€nû¬<2 #- _wandbruntime‚9352  _runtime‚93522C[² '¡#ƒN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11656124929536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11656211402240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100975137 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100975137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86458274 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100985128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443404 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187433411 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11532360932352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11532360932352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86458276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19096163 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364960502286336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42533082 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660196436055431 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1648682007633920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443404 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376616713688576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100985128 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1668740965493760 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1657208604561408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚897 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10899 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1648682024673280 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1657208604561408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532019586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376616608603648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660196793644793 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86458276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23436916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23436644 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78434969401417e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42532807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19096166 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚9352.707927557 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1668740965493760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1668740607108096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514428421511Šž'¢#ˆN TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376616608603648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660196793644793 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86458276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23436916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23436644 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78434969401417e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42532807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19096166 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚9352.707927557 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1668740965493760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1668740607108096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514428421511 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11656124929536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11656211402240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100975137 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100975137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86458274 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100985128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443404 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187433411 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11532360932352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11532360932352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86458276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19096163 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364960502286336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42533082 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660196436055431 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1648682007633920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443404 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376616713688576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100985128 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1668740965493760 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1657208604561408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚897 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10899 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1648682024673280 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1657208604561408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532019586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983Œ5v2£#- _wandbruntime‚9352  _runtime‚9352ÖˆÁ ¤#† /loss/FRt2/detection_loss‚1.5189353546096949 1loss/SRqqt2/detection_loss‚0.5824920467746331 /loss/FBt1/detection_loss‚0.6565129676567657 _runtime‚9352.710704398 0loss/FRt1/assignment_loss‚1.5408451880137433 2loss/FBt2/assignment_loss‚0.008228197541612506 % _timestamp‚1.7843496940173855e+09 _step‚898 /loss/FBt2/detection_loss‚0.4731004123006382 epoch‚27 2loss/SRqqt1/assignment_loss‚0.5147317655528835 2loss/SRqqt2/assignment_loss‚0.4583035649085548 3loss/FBt1/assignment_loss‚8.649854658833484e-05 /loss/FRt1/detection_loss‚0.5531278309632036 1loss/SRqqt1/detection_loss‚0.9209083944998888 0loss/FRt2/assignment_loss‚0.6330493181792612 %loss/total_loss‚7.860321539547468 trainer/global_step‚10899ë$ß4¥#‹ 0loss/FRt1/assignment_loss‚1.5408451880137433 2loss/FBt2/assignment_loss‚0.008228197541612506 % _timestamp‚1.7843496940173855e+09 _step‚898 /loss/FBt2/detection_loss‚0.4731004123006382 epoch‚27 2loss/SRqqt1/assignment_loss‚0.5147317655528835 2loss/SRqqt2/assignment_loss‚0.4583035649085548 3loss/FBt1/assignment_loss‚8.649854658833484e-05 /loss/FRt1/detection_loss‚0.5531278309632036 1loss/SRqqt1/detection_loss‚0.9209083944998888 0loss/FRt2/assignment_loss‚0.6330493181792612 %loss/total_loss‚7.860321539547468 trainer/global_step‚10899 /loss/FRt2/detection_loss‚1.5189353546096949 1loss/SRqqt2/detection_loss‚0.5824920467746331 /loss/FBt1/detection_loss‚0.6565129676567657 _runtime‚9352.710704398‚úÞ¼)“¦#‚(:ˆ …ˆìÒ«íñ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€F‘x§#‚(:é …ˆìÒíÞ“– disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.dm-0.in‚16433.95703125" disk.n”£*|vme0n1.in‚16433.95703125 disk.rbd1.out‚ 33.9541015625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 285.98828125$ disk.nvme0c0n1.in‚16433.95703125' disk.nvme0c0n1.out‚26983.0283203125 disk.drbd2165.out‚0) proc.memory.percent‚16.77250862121582 network.recv‚ 2.536783e+06( disk./.usagePercent‚39.4887197755457 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0$ memory_percent‚97.10445404052734 network.sent‚ 1.7393736e+07" disk.dm-0.out‚26983.0283203125% disk.nvme0n1.out‚26983.0283203125 disk.sda3.out‚ 285.98828125 disk.dm-1.out‚ 285.98828125% proc.memory.availableMB‚ 474.40625" proc.memory.rssMB‚ 2748.0078125 cpu‚26.181428358755465$ disk./.usageGB‚588.4104614257812 disk.sda2.in‚0 disk.rbd0.out‚0ã"·£¨#‚(:’ ”ˆìÒÎÖ¨• gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.735599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%SÏÐù©#‚(:î ”ˆìÒöÁÜ–( disk.nvme0c0n1.out‚26986.79052734375# disk./.usageGB‚588.410472869873 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚16501.01171875 disk.sda3.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125 network.sent‚ 1.7456808e+07 network.recv‚ 2.541084e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚26986.79052734375 disk.sda3.out‚ 286.44140625) proc.memory.percent‚16.77250862121582* disk./.usagePercent‚39.488720543568306$ disk.nvme0c0n1.in‚16501.01171875" disk.nvme0n1.in‚16501.01171875( proc.memory.availableMB‚ 474.69140625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 34.0166015625 disk.sda1.in‚0 disk.dm-1.out‚ 286.44140625 disk.sda.out‚ 286.44140625 cpu‚26.18124205168183# disk.dm-0.out‚26986.79052734375 disk.sda.in‚ 0.1328125# memory_percent‚97.1027135848999 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚ 2.01953125 disk.sda2.in‚0ŒÐZ“ª#‚(:ˆ £ˆìÒ—ë´“ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚264.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûw!cê«#‚(:ß £ˆìÒöë– disk.dm-1.out‚ 286.90625& disk.nvme0c0n1.out‚26989.451171875 disk.dm-0.in‚16501.01171875 disk.sda2.in‚0 disk.sda3.out‚ 286.90625 disk.rbd0.out‚0) disk./.usagePercent‚39.48872028756077$ disk.nvme0c0n1.in‚16501.01171875 disk.rbd1.out‚ 34.0166015625 disk.sda1.in‚0$ memory_percent‚97.10421562194824 cpu‚26.18103233132721 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚16501.01171875 disk.rbd1.in‚ 2.01953125 network.sent‚ 1.7461931e+07! disk.dm-0.out‚26989.451171875 disk.rbd0.in‚0' proc.memory.availableMB‚ 474.4453125" proc.memory.rssMB‚ 2748.0078125) proc.memory.percent‚16.77250862121582 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125 network.recv‚ 2.544711e+06 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.out‚ 286.90625$ disk.nvme0n1.out‚26989.451171875 disk.sda1.out‚0$ disk./.usageGB‚588.4104690551758>bV2¬#- _wandbruntime‚9392  _runtime‚9392¼ëÇÈ­# & lr-AdamW/pg2‚0.0001976465000451278 trainer/global_step‚10949 % _timestamp‚1.7843497342517471e+09 _runtime‚9392.945309351 _step‚899 & lr-AdamW/pg1‚0.0001976465000451278ªÎŽÍ®#Ç trainer/global_step‚10949 % _timestamp‚1.7843497342517471e+09 _runtime‚9392.945309351 _step‚899 & lr-AdamW/pg1‚0.0001976465000451278 & lr-AdamW/pg2‚0.0001976465000451278ƒJŸ2¯#- _wandbruntime‚9392  _runtime‚9392üðT(°#ûO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11564374404587 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1664423397974016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11564033854697 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86832242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19181671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378304976087552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1664423397974016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843497342527785e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86832244 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101420537 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚10949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101430514 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1676005100434944 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1664423415013376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19181668 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378304873092608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11706380250624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1655861047595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1676005458813440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101430514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23539299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11706464633344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86832244 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088¤m4(±#€P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86832244 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101420537 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚10949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101430514 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1676005100434944 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1664423415013376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19181668 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378304873092608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11706380250624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1655861047595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1676005458813440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101430514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23539299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11706464633344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86832244 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11564374404587 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1664423397974016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11564033854697 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86832242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19181671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378304976087552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1664423397974016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843497342527785e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0„e6 è2²#- _wandbruntime‚9393  _runtime‚9393#Òä²ä³#„N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1656007541227520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101439580 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101439580 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1676153712477184 trainer/global_step‚10949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188279456 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11583056434176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1664570656043008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42724812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11707489064960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23541149 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86839876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378339429277696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11707408883712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86839876 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1667572586135939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1667572943725301 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1676153354091520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11583056434176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19183413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366631921600512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86839874 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843497348668203e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9393.560619001 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86839874 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1664570656043008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23541396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11565385458421 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11582715087872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101429589 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366631940212736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269463 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1676153712477184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1656007558266880 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42724562 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19183416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101429589 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366631940212736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269463 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1676153712477184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1656007558266880 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42724562 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19183416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1676153712477184 trainer/global_step‚10949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188279456 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11583056434176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1664570656043008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42724812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11707489064960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23541149 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86839876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378339429277696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11707408883712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86839876 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1667572586135939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1667572943725301 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1676153354091520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11583056434176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19183413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366631921600512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86839874 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843497348668203e+09 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9393.560619001 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86839874…ž÷fò2µ#- _wandbruntime‚9393  _runtime‚9393ŽxÔ ¶#‡ 0loss/FRt1/assignment_loss‚1.4550725965929372 2loss/FBt2/assignment_loss‚0.027660977462591076 epoch‚27 1loss/SRqqt2/detection_loss‚0.6023678909811176 /loss/FBt1/detection_loss‚0.5849159858594446 trainer/global_step‚10949 _runtime‚9393.563438093 _step‚902 0loss/FRt2/assignment_loss‚0.6018318588216903 3loss/SRqqt2/assignment_loss‚0.46032185472689197 /loss/FRt1/detection_loss‚0.6001929592129326 /loss/FRt2/detection_loss‚1.4914574589027039 /loss/FBt2/detection_loss‚0.3641933982231779 % _timestamp‚1.7843497348701293e+09 2loss/SRqqt1/assignment_loss‚0.5338101715870428 3loss/FBt1/assignment_loss‚0.0007537273468569308 1loss/SRqqt1/detection_loss‚0.7277876446322852 %loss/total_loss‚7.450366524349672olÇ£·#Œ 0loss/FRt1/assignment_loss‚1.4550725965929372 2loss/FBt2/assignment_loss‚0.027660977462591076 epoch‚27 1loss/SRqqt2/detection_loss‚0.6023678909811176 /loss/FBt1/detection_loss‚0.5849159858594446 trainer/global_step‚10949 _runtime‚9393.563438093 _step‚902 0loss/FRt2/assignment_loss‚0.6018318588216903 3loss/SRqqt2/assignment_loss‚0.46032185472689197 /loss/FRt1/detection_loss‚0.6001929592129326 /loss/FRt2/detection_loss‚1.4914574589027039 /loss/FBt2/detection_loss‚0.3641933982231779 % _timestamp‚1.7843497348701293e+09 2loss/SRqqt1/assignment_loss‚0.5338101715870428 3loss/FBt1/assignment_loss‚0.0007537273468569308 1loss/SRqqt1/detection_loss‚0.7277876446322852 %loss/total_loss‚7.450366524349672†v4,퓸#‚(:ˆ ²ˆìÒü×á“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚240.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.1592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¶g‡ô¹#‚(:é ²ˆìÒ›Š÷•# disk./.usageGB‚588.410400390625 disk.sda3.in‚ 0.1328125$ disk.nvme0c0n1.in‚16501.01171875 disk.dm-0.in‚16501.01171875# disk.dm-0.out‚26992.91064453125 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125 network.sent‚ 1.7467501e+07( disk.nvme0c0n1.out‚26992.91064453125 disk.sda2.out‚0& disk.nvme0n1.out‚26992.91064453125( proc.memory.availableMB‚ 474.19921875 network.recv‚ 2.548139e+06 disk.sda.in‚ 0.1328125 disk.sda.out‚ 287.171875" disk.nvme0n1.in‚16501.01171875) proc.memory.percent‚16.77250862121582 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 287.171875 disk.rbd1.out‚ 34.0791015625$ memory_percent‚97.10571765899658 cpu‚26.180882242196088) disk./.usagePercent‚39.48871567942515 disk.dm-1.out‚ 287.171875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2748.0078125ŠN$ò’º#‚(:‡ ÁˆìÒæåË“ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚264.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.935 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–.mæó»#‚(:è ÁˆìÒÁÌý• disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.in‚0& disk.nvme0n1.out‚26995.71044921875 disk.sda3.in‚ 0.1328125# memory_percent‚97.1057653427124) proc.memory.percent‚16.77250862121582 disk.sda2.out‚0 disk.sda3.out‚ 287.515625( proc.memory.availableMB‚ 474.19140625 network.recv‚ 2.552892e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚26.18064770520207 proc.cpu.threads‚17 network.sent‚ 1.7530026e+07$ disk./.usageGB‚588.4108543395996 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚16501.01171875 disk.dm-0.in‚16501.01171875 disk.rbd1.in‚ 2.01953125" proc.memory.rssMB‚ 2748.0078125( disk.nvme0c0n1.out‚26995.71044921875# disk.dm-0.out‚26995.71044921875 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 34.0791015625) disk./.usagePercent‚39.48874614432173 disk.dm-1.out‚ 287.515625 disk.sda.out‚ 287.515625" disk.nvme0n1.in‚16501.01171875×}–‡2¼#- _wandbruntime‚9433  _runtime‚9433OÐÖʽ#Ä _step‚903 ' lr-AdamW/pg1‚0.00019586082404090448 ' lr-AdamW/pg2‚0.00019586082404090448 trainer/global_step‚10999 % _timestamp‚1.7843497752787583e+09 _runtime‚9433.975467601†ñCϾ#É _runtime‚9433.975467601 _step‚903 ' lr-AdamW/pg1‚0.00019586082404090448 ' lr-AdamW/pg2‚0.00019586082404090448 trainer/global_step‚10999 % _timestamp‚1.7843497752787583e+09‡£ Í2¿#- _wandbruntime‚9433  _runtime‚9433ùé¡(À#ûO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚10999 _runtime‚9433.976426752 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1671785449455616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87213842 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11757843903488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87213842 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101874989 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368269949380608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843497752827423e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101874989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23643887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87213844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42913077 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11757759520768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87213844 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1683418205796864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380027690289152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674801214490853 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11632739301888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1683418205796864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088831 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23644156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380027793284096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101884966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1671785449455616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19268921 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1674801572080103 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1671785466494976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632397962752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189098810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368269930768384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1683417847418368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42912805 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632397962752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1663186581188608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976™¾Á#€P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380027690289152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674801214490853 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11632739301888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1683418205796864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088831 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23644156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380027793284096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101884966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1671785449455616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19268921 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1674801572080103 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1671785466494976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632397962752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189098810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368269930768384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1683417847418368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42912805 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632397962752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1663186581188608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pooláq¥ÿ.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚10999 _runtime‚9433.976426752 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1671785449455616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87213842 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11757843903488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87213842 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101874989 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368269949380608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843497752827423e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101874989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23643887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87213844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42913077 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11757759520768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87213844 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1683418205796864 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ˆàÔ¾±2Â#- _wandbruntime‚9434  _runtime‚9434ž”’ß 'Ã#„N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101884041 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101884041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1663333074821120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87221474 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11633751936000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11633751936000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105515 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42916857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368303378139136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87221476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19270666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚905 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1683566459460608 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1671932707524608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚92502016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11615661395327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380062254755840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1663333091860480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87221476 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1671932690485248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23645942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42916605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ò â]'Ä#‰N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42916605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101884041 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101884041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1663333074821120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87221474 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11633751936000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11633751936000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105515 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42916857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368303378139136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87221476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19270666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚905 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1683566459460608 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1671932707524608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚92502016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11615661395327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380062254755840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1663333091860480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87221476 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1671932690485248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23645942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1690830594401792 ÚVôˆ"TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11049 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚9474.790377651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102329441 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924883 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11683434803712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679147500937216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11683093464576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11665607378403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1690830952780288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11809037040128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934862 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1690830594401792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11809134005760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1682177364571361 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1670512114782208Žm9-(Ó#ÿO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924883 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11683434803712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679147500937216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11683093464576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11665607378403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1690830952780288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11809037040128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934862 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1690830594401792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11809134005760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1682177364571361 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1670512114782208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1690830594401792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11049 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚9474.790377651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102329441 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0Œ,ÆØ2Ô#- _wandbruntime‚9475  _runtime‚9475\œGY 'Õ#…N FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1670658608414720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1690979206444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1679294741966848 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1679294759006208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚909 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1679294759006208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1670658625454080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1682325243886317 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1682324886296955 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189951560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚9475.406482366 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1679294741966848 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87603076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843498167126367e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941567 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43108678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1690979206444032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19357913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23750486 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810155432448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381784868510208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚11049 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87603074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87603074 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43108399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19357916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87603076 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102348484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369974797453312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1690978848058368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843498167126367e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941567 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43108678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1690979206444032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19357913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23750486 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810155432448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381784868510208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚11049 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87603074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87603074 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43108399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19357916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87603076 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102348484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.smaýll_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369974797453312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1690978848058368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1670658608414720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1690979206444032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1679294741966848 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1679294759006208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚909 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1679294759006208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1670658625454080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1682325243886317 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1682324886296955 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088n[_’2×#- _wandbruntime‚9475  _runtime‚9475$WøØ#‰ 2loss/SRqqt2/assignment_loss‚0.4533074391679997 /loss/FRt2/detection_loss‚1.4496240650855985 1loss/SRqqt1/detection_loss‚0.8898044647466683 /loss/FBt1/detection_loss‚0.4612856485225536 0loss/FRt2/assignment_loss‚0.6147505527098777 2loss/SRqqt1/assignment_loss‚0.5674006297924552 4loss/FBt1/assignment_loss‚0.00011213524061343744 1loss/SRqqt2/detection_loss‚0.6415118571297315 % _timestamp‚1.7843498167154002e+09 _runtime‚9475.408704911 0loss/FRt1/assignment_loss‚1.5498771184399014 4loss/FBt2/assignment_loss‚0.00024318657448730178 /loss/FBt2/detection_loss‚0.5406518197814589 epoch‚27 _step‚910 /loss/FRt1/detection_loss‚0.5384457246339852 %loss/total_loss‚7.707014641825331 trainer/global_step‚11049@«£ Ù#Ž 2loss/SRqqt2/assignment_loss‚0.4533074391679997 /loss/FRt2/detection_loss‚1.4496240650855985 1loss/SRqqt1/detection_loss‚0.8898044647466683 /loss/FBt1/detection_loss‚0.4612856485225536 0loss/FRt2/assignment_loss‚0.6147505527098777 2loss/SRqqt1/assignment_loss‚0.5674006297924552 4loss/FBt1/assignment_loss‚0.00011213524061343744 1loss/SRqqt2/detection_loss‚0.6415118571297315 % _timestamp‚1.7843498167154002e+09 _runtime‚9475.408704911 0loss/FRt1/assignment_loss‚1.5498771184399014 4loss/FBt2/assignment_loss‚0.00024318657448730178 /loss/FBt2/detection_loss‚0.5406518197814589 epoch‚27 _step‚910 /loss/FRt1/detection_loss‚0.5384457246339852 %loss/total_loss‚7.707014641825331 trainer/global_step‚11049Žì.w·Ú#‚(:’ ýˆìÒâÑ…” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.604' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.72079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼¾¹ïõÛ#‚(:ê ýˆìÒî„™• network.sent‚ 1.7607882e+07) disk./.usagePercent‚39.48874512029159 disk.sda2.out‚0$ memory_percent‚97.10586071014404$ disk./.usageGB‚588.4108390808105 disk.rbd0.out‚0 disk.dm-1.out‚ 288.6171875 disk.sda.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda3.out‚ 288.6171875 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 34.2041015625 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.dm-0.out‚27013.2021484375 disk.sda2.in‚0" disk.nvme0n1.in‚16562.76171875% disk.nvme0n1.out‚27013.2021484375) proc.memory.percent‚16.77250862121582 network.recv‚ 2.567146e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚16562.76171875' disk.nvme0c0n1.out‚27013.2021484375 disk.sda.out‚ 288.6171875( proc.memory.availableMB‚ 474.17578125 disk.dm-0.in‚16562.76171875 disk.sda3.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125 cpu‚26.1794860796038621—bQ‘Ü#‚(:† Œ‰ìÒÒÛÞ“ gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚253.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›´âðïÝ#‚(:ä Œ‰ìÒƒ·‚– network.recv‚ 2.571361e+06 disk.drbd2165.in‚0# proc.memory.availableMB‚473.625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 16614.0546875 disk.dm-0.in‚ 16614.0546875& disk.nvme0n1.out‚27025.41552734375) proc.memory.percent‚16.77250862121582$ disk./.usageGB‚588.4073028564453 disk.rbd1.out‚ 34.2666015625# disk.dm-0.out‚27025.41552734375" proc.memory.rssMB‚ 2748.0078125 network.sent‚ 1.7671182e+07 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27025.41552734375! disk.nvme0n1.in‚ 16614.0546875 disk.sda1.out‚0 disk.dm-1.out‚ 290.5859375 disk.drbd2165.out‚0 disk.sda.out‚ 290.5859375 disk.sda2.in‚0$ memory_percent‚97.10922241210938 cpu‚26.17932987583157) disk./.usagePercent‚39.48850780130735 disk.sda3.out‚ 290.5859375 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0êçàÞ#‚(:’ ›‰ìÒÿå¾” gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.792' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.55839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G0Iûß#‚(:𠛉ìÒüýŠ– disk.drbd2165.out‚0) proc.memory.percent‚16.77250862121582 disk.sda.in‚ 0.1328125 disk.sda.out‚ 291.11328125 disk.sda1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 473.62109375 disk.sda3.in‚ 0.1328125( disk.nvme0c0n1.out‚27029.63037109375 network.recv‚ 2.574826e+06# disk.dm-0.out‚27029.63037109375 disk.sda2.out‚0 disk.sda3.out‚ 291.11328125 disk.rbd0.in‚0 disk.dm-0.in‚16616.69921875 disk.dm-1.out‚ 291.11328125$ disk.nvme0c0n1.in‚16616.69921875 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 1.7676324e+07) disk./.usagePercent‚39.48850600925461" disk.nvme0n1.in‚16616.69921875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125" proc.memory.rssMB‚ 2748.0078125 cpu‚26.179047508049546$ disk./.usageGB‚588.4072761535645& disk.nvme0n1.out‚27029.63037109375 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 34.2666015625$ memory_percent‚97.10924625396729 proc.cpu.threads‚17äT L2à#- _wandbruntime‚9515  _runtime‚9515HU{NÉá#à _runtime‚9515.804359011 _step‚911 ' lr-AdamW/pg1‚0.00019229511233987852 ' lr-AdamW/pg2‚0.00019229511233987852 trainer/global_step‚11099 $ _timestamp‚1.784349857110774e+09özÎâ#È ' lr-AdamW/pg2‚0.00019229511233987852 trainer/global_step‚11099 $ _timestamp‚1.784349857110774e+09 _runtime‚9515.804359011 _step‚911 ' lr-AdamW/pg1‚0.00019229511233987852#2ã#- _wandbruntime‚9515  _runtime‚9515Õï(ä#úO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11715883315421 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1698243341385216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚11099 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87977044 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1698243699763712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1686509552418816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102783893 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1686509569458176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190770914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87977042 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1686509552418816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19443421 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87977042 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1686509569458176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383473100475904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19443418 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1677837631336448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760935 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1689553514651869 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43294684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87977044 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23851533 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784349857111912e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1698243341385216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11733788966400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224oÕÅ™%å#ÿO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1698243699763712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1686509552418816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102783893 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1686509569458176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190770914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87977042 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1686509552418816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19443421 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87977042 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1686509569458176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383473100475904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19443418 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1677837631336448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760935 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1689553514651869 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43294684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87977044 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23851533 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784349857111912e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1698243341385216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11733788966400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11715883315421 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batchB€ð½l_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1698243341385216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚11099 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87977044 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872|¦›ï2æ#- _wandbruntime‚9516  _runtime‚9516Ë—‡Š 'ç#…N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102802936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383507433081856 trainer/global_step‚11099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1698391953427456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190777619 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87984676 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371646235379712 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1677984159047680 _runtime‚9516.420416457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777619 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787612 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11716894369143 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1698391953427456 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1686656810487808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102792945 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1689701036377463 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19445166 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11861197702144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87984674 _step‚913 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371646253991936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102792945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1689701393966825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43298755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19445163 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1686656810487808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43298489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11717234919145 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23853589 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11735142939648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11734801593344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23853326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0š¯oô'è#ŠN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787612 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11716894369143 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1698391953427456 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1686656810487808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102792945 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1689701036377463 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19445166 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11861197702144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87984674 _step‚913 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371646253991936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102792945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1689701393966825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43298755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19445163 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1686656810487808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43298489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11717234919145 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23853589 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11735142939648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11734801593344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23853326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102802936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383507433081856 trainer/global_step‚11099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1698391953427456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190777619 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87984676 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371646235379712 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1677984159047680 _runtime‚9516.420416457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777619 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002‘™ T2é#- _wandbruntime‚9516  _runtime‚95161Nf ê#† /loss/FBt1/detection_loss‚0.5340178928818204 _runtime‚9516.423272178 _step‚914 $loss/total_loss‚7.51652855041576 % _timestamp‚1.7843498577299547e+09 0loss/FRt2/assignment_loss‚0.5756569211347116 3loss/FBt1/assignment_loss‚7.908107503507011e-06 1loss/SRqqt1/detection_loss‚0.8559190141089575 3loss/SRqqt1/assignment_loss‚0.42429304342798607 /loss/FRt1/detection_loss‚0.5359984363650254 1loss/SRqqt2/detection_loss‚0.6158974529187218 epoch‚27 trainer/global_step‚11099 /loss/FBt2/detection_loss‚0.3696624790448195 0loss/FRt1/assignment_loss‚1.5475675157997066 2loss/SRqqt2/assignment_loss‚0.4719988851338104 2loss/FBt2/assignment_loss‚0.009844948697464464 /loss/FRt2/detection_loss‚1.5756640527952328Ð)ÏEë#‹ /loss/FRt2/detection_loss‚1.5756640527952328 /loss/FBt1/detection_loss‚0.5340178928818204 _runtime‚9516.423272178 _step‚914 $loss/total_loss‚7.51652855041576 % _timestamp‚1.7843498577299547e+09 0loss/FRt2/assignment_loss‚0.5756569211347116 3loss/FBt1/assignment_loss‚7.908107503507011e-06 1loss/SRqqt1/detection_loss‚0.8559190141089575 3loss/SRqqt1/assignment_loss‚0.42429304342798607 /loss/FRt1/detection_loss‚0.5359984363650254 1loss/SRqqt2/detection_loss‚0.6158974529187218 epoch‚27 trainer/global_step‚11099 /loss/FBt2/detection_loss‚0.3696624790448195 0loss/FRt1/assignment_loss‚1.5475675157997066 2loss/SRqqt2/assignment_loss‚0.4719988851338104 2loss/FBt2/assignment_loss‚0.009844948697464464’TÅzäžì#‚(:“ ª‰ìÒ᧴“ gpu.0.gpu‚27.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.735599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h$‹ìí#‚(:á ª‰ìÒî¼— disk.sda1.in‚0$ memory_percent‚97.10915088653564" proc.memory.rssMB‚ 2748.0078125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚16616.69921875 disk.dm-0.in‚16616.69921875" disk.nvme0n1.in‚16616.69921875( proc.memory.availableMB‚ 473.63671875 network.recv‚ 2.577931e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0! disk.nvme0n1.out‚ 27037.734375 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 27037.734375 disk.rbd1.out‚ 34.3291015625 cpu‚26.178995996477255 network.sent‚ 1.768119e+07* disk./.usagePercent‚39.489021864436104 disk.dm-1.out‚ 291.73828125 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚588.4149627685547 proc.cpu.threads‚17 disk.dm-0.out‚ 27037.734375 disk.sda2.in‚0 disk.sda3.out‚ 291.73828125) proc.memory.percent‚16.77250862121582 disk.sda.out‚ 291.73828125ŽhÊ“î#‚(:ˆ ¹‰ìÒÍÎå” gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚224.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.9356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îsóï#‚(:è ¹‰ìÒ§„Þ• proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0# memory_percent‚97.1081018447876 disk.dm-1.out‚ 292.1015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚16616.69921875" proc.memory.rssMB‚ 2748.0078125 network.sent‚ 1.7744316e+07$ disk.nvme0c0n1.in‚16616.69921875 disk.sda3.out‚ 292.1015625 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.percent‚16.77250862121582" disk.dm-0.out‚27041.3251953125 disk.sda.out‚ 292.1015625 disk.rbd1.out‚ 34.3291015625 disk.dm-1.in‚ 0.1328125' disk.nvme0c0n1.out‚27041.3251953125% disk.nvme0n1.out‚27041.3251953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 473.80859375 network.recv‚ 2.582373e+06 disk.dm-0.in‚16616.69921875 disk.sda.in‚ 0.1328125) disk./.usagePercent‚39.48903031268473$ disk./.usageGB‚588.4150886535645 cpu‚26.17882780259989#“±p“ð#‚(:ˆ ȉìÒ¯õî“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜDS&ùñ#‚(:î ȉìÒ€åÖ– disk.dm-0.in‚16616.69921875 disk.rbd1.out‚ 34.4541015625 disk.sda1.out‚0( proc.memory.availableMB‚ 473.55859375" proc.memory.rssMB‚ 2748.0078125 proc.cpu.threads‚17 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚27048.41552734375 disk.sda2.out‚0) proc.memory.percent‚16.77250862121582 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 292.23046875" disk.nvme0n1.in‚16616.69921875 disk.sda3.out‚ 292.23046875 disk.rbd1.in‚ 2.01953125& disk.nvme0n1.out‚27048.41552734375) disk./.usagePercent‚39.48902544854159$ disk./.usageGB‚588.4150161743164 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0# disk.dm-0.out‚27048.41552734375 network.recv‚ 2.585647e+06$ disk.nvme0c0n1.in‚16616.69921875 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚26.1787774190823$ memory_percent‚97.10962772369385 network.sent‚ 1.7749294e+07 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 292.230468751 BQ2ò#- _wandbruntime‚9556  _runtime‚9556ü|PÍÊó#Ä trainer/global_step‚11149 % _timestamp‚1.7843498978916478e+09 _runtime‚9556.585236323 _step‚915 ' lr-AdamW/pg1‚0.00019051530566207523 ' lr-AdamW/pg2‚0.00019051530566207523vþÃÏô#É trainer/global_step‚11149 % _timestamp‚1.7843498978916478e+09 _runtime‚9556.585236323 _step‚915 ' lr-AdamW/pg1‚0.00019051530566207523 ' lr-AdamW/pg2‚0.00019051530566207523“ÊÌ(K2õ#- _wandbruntime‚9556  _runtime‚9556†„Ëáö#úO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596987 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385195591133184 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1685163164930048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88358644 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19530671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43485349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373284244547584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚916 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1696929664732377 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1705656446747136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1696930022321627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103238345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705656088368640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1705656088368640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373284263159808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23954945 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693871603900416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚119’èk²11327973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88358644 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784349897895512e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103248322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23954681 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385195483943936 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1685163181969408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11784825807360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191606966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103238345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43485616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11766840352219 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103248322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693871603900416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11784825807360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11784484468224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚32‘ê$(÷#ÿO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚9556.589251625 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693871620939776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19530668 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚11149 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11784484468224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693871620939776 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705656088368640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1705656088368640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373284263159808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23954945 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693871603900416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11911327973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88358644 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784349897895512e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103248322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23954681 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385195483943936 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1685163181969408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11784825807360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191606966 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103238345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43485616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11766840352219 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103248322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693871603900416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11784825807360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11784484468224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596987 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385195591133184 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1685163164930048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88358644 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19530671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43485349 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373284244547584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚916 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1696929664732377 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1705656446747136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1696930022321627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103238345 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88366276 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385230040538112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1697077186457971 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚917 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1685309675601920 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 9557.3884242 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694018844930048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11785497095168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43489344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19532416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373317673306112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613671 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43489072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88366274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19532413 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23956928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373317691918336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694018844930048 $ _timestamp‚1.784349898694654e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705804342025216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103257388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11912348619776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191613671 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1685309692641280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11767851406053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103257388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23956659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0„¨¹°ú#‡N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11767510856051 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191623664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11785838441472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694018861969408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1697077544047333 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694018844930048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11785497095168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43489344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19532416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373317673306112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613671 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43489072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88366274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19532413 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23956928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373317691918336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694018844930048 $ _timestamp‚1.784349898694654e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705804342025216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103257388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11912348619776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191613671 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1685309692641280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11767851406053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103257388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23956659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMoni®¥ÿ!Žtor.on_train_batch_end/active.small_pool.freed‚ 103247397 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694018861969408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88366276 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385230040538112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1697077186457971 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚917 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1685309675601920 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 9557.3884242 Q>2…$- _wandbruntime‚9597  _runtime‚9597%–‡(†$ûO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713068835352064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88740242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11835521309184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24058073 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386918039203328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43675724 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11962338117120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1701233655382016 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713069193730560 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713069193730560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713068835352064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103702774 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1692488698523648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386917929916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1692488715563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1701233672421376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11835179970048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1701233655382016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192433039 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88740242 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374955682473984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103692797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11962247442944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376ÑÆÛc(‡$€P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88740242 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374955682473984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103692797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11962247442944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713068835352064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88740242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11835521309184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24058073 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386918039203328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43675724 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11962338117120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1701233655382016 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713069193730560 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713069193730560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713068835352064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103702774 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1692488698523648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386917929916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1692488715563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1701233672421376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11835179970048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1701233655382016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192433039 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438˜ ƒR2ˆ$- _wandbruntime‚9598  _runtime‚9598ÿ”ûz‰$…N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1704453694127841 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1701380896411648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459716 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103701849 _runtime‚9598.198442868 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103711840 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386952484135936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103701849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43679551 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11836533943296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1692635209195520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24060154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11836192596992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713217089008640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11963354291200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚921 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449723 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1701380896411648 J@DeviceStatsMonitor.on_train_ ôa7‘batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374989111232512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43679820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374989129844736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11836192596992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103711840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24059888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11963261527040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1701380913451008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19619666 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459716 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386952372759552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88747876 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88747874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0‚M…Ê'Š$ŠN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103711840 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386952484135936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103701849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43679551 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11836533943296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1692635209195520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24060154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11836192596992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713217089008640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11963354291200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚921 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449723 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1701380896411648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374989111232512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43679820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374989129844736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11836192596992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103711840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24059888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11963261527040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1701380913451008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19619666 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459716 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386952372759552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88747876 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88747874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1704453694127841 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1701380896411648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459716 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103701849 _runtime‚9598.198442868™xôË×2‹$- _wandbruntime‚9598  _runtime‚9598ëü¦¡Œ$‚ 0loss/FRt1/assignment_loss‚1.4730854167004501 2loss/FBt1/assignment_loss‚8.66204881374344e-05 0loss/FBt2/assignment_loss‚0.0202423075991431 /loss/FRt2/detection_loss‚1.5593196103812532 .loss/FBt2/detection_loss‚0.476071217380788 _runtime‚9598.201934019 2loss/SRqqt1/assignment_loss‚0.5617801309352205 _step‚922 2loss/SRqqt2/assignment_loss‚0.4161680610181594 1loss/SRqqt1/detection_loss‚0.7650906032322281 1loss/SRqqt2/detection_loss‚0.6696088515168971 epoch‚27 % _timestamp‚1.7843499395086427e+09 0loss/FRt2/assignment_loss‚0.6549523034007694 /loss/FRt1/detection_loss‚0.5880590658991091 /loss/FBt1/detection_loss‚0.3749454077333212 %loss/total_loss‚7.559409596285477 trainer/global_step‚11199ç’¿J $‡ 0loss/FRt1/assignment_loss‚1.4730854167004501 2loss/FBt1/assignment_loss‚8.66204881374344e-05 0loss/FBt2/assignment_loss‚0.0202423075991431 /loss/FRt2/detection_loss‚1.5593196103812532 .loss/FBt2/detection_loss‚0.476071217380788 _runtime‚9598.201934019 2loss/SRqqt1/assignment_loss‚0.5617801309352205 _step‚922 2loss/SRqqt2/assignment_loss‚0.4161680610181594 1loss/SRqqt1/detection_loss‚0.7650906032322281 1loss/SRqqt2/detection_loss‚0.6696088515168971 epoch‚27 % _timestamp‚1.7843499395086427e+09 0loss/FRt2/assignment_loss‚0.6549523034007694 /loss/FRt1/detection_loss‚0.5880590658991091 /loss/FBt1/detection_loss‚0.3749454077333212 %loss/total_loss‚7.559409596285477 trainer/global_step‚11199š9f›ÐžŽ$‚(:“ õ‰ìÒú©” gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.094' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.218799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­üéó$‚(:è õ‰ìÒ‘س– disk.sda1.in‚0 proc.cpu.threads‚17! disk.dm-0.out‚27063.146484375" disk.nvme0n1.in‚16677.27734375$ memory_percent‚97.11003303527832) proc.memory.percent‚16.77250862121582$ disk.nvme0c0n1.in‚16677.27734375" proc.memory.rssMB‚ 2748.0078125$ disk./.usageGB‚588.4149017333984 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚16677.27734375 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 294.14453125 disk.rbd1.in‚ 2.01953125 network.sent‚ 1.7822305e+07 network.recv‚ 2.596821e+06 disk.sda.out‚ 294.14453125$ disk.nvme0n1.out‚27063.146484375 cpu‚26.17851453652618) disk./.usagePercent‚39.48901776831556 disk.dm-1.out‚ 294.14453125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚27063.146484375 disk.sda.in‚ 0.1328125' proc.memory.availableMB‚ 473.4921875 disk.rbd0.in‚0 disk.rbd1.out‚ 34.5166015625GoJR“$‚(:ˆ „ŠìÒøÐâ“ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚284.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ§%çú‘$‚(:ï „ŠìÒçÊÇ•( disk./.usagePercent‚39.4890382489183 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.23837375640869 network.recv‚ 2.601099e+06$ disk./.usageGB‚588.4152069091797 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 294.44140625 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚16677.27734375 disk.dm-0.in‚16677.27734375 disk.sda2.out‚0 cpu‚26.18202650907493 disk.sda1.in‚0 disk.sda.out‚ 294.44140625( proc.memory.availableMB‚ 452.46484375" proc.memory.rssMB‚ 2768.9453125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚27069.69384765625# disk.dm-0.out‚27069.69384765625 disk.rbd1.out‚ 34.5791015625* proc.memory.percent‚16.900300979614258 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 294.44140625" disk.nvme0n1.in‚16677.27734375& disk.nvme0n1.out‚27069.69384765625 proc.cpu.threads‚17 network.sent‚ 1.7885352e+07œžI2’$- _wandbruntime‚9616  _runtime‚9616‹¬?¬{3“$ñz 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2681746482849121 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9956162571907043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02637808583676815 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7446609735488892 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3250144422054291 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7164120674133301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1347973495721817 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037072885781526566 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.25982481241226196 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4993777573108673 jet/accuracy_5_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7071945667266846 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21187171339988708 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28531524538993835 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2094418704509735 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22456161677837372 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8763028979301453 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8759260773658752 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_3_of_5‚1 epoch‚27 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22433602809906006 *jet/accuracy_2_of_4‚0.7530478835105896 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28750738501548767 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8425170183181763 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675925374031067 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2766534388065338 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04125630110502243 *jet/accuracy_1_of_4‚0.9846457242965698 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8006976842880249 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.389280766248703 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6593807935714722 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7828143835067749 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5131988525390625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7809864282608032 )loss/val_total_loss‚7.647209171172678 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37834885716438293 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21089054644107819 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01985609158873558 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6578532457351685 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23521658778190613 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_2_of_2‚0.13075630366802216 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6898267269134521 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/sensitivity‚0.6029893755912781 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7116103172302246 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02901482954621315 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +particle/specificity‚0.9006449580192566 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7668827772140503 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8267384767532349 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.098396435379982 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968291521072388 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6133191585540771 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2356867790222168 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8238095045089722 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02621140331029892 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6593807935714722 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07447560876607895 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7903681993484497 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7924433946609497 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11641217768192291 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6900812983512878 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.76036536693573 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23080816864967346 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.060833342373371124 (particle/accuracy‚0.8367165327072144 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27809709310531616 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04650595784187317 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5827711820602417 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22919043898582458 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2652595043182373 /particle/accuracy_4_of_4‚0.9539281129837036 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31657788157463074 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1512133926153183 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7333460450172424 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8763028979301453 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7051434516906738 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6751008629798889 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23650391399860382 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226568102836609 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6406713128089905 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24201788008213043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7069951891899109 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.732225775718689 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.715948760509491 particle/accuracy_1_of_1‚1 trainer/global_step‚11199 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27576276659965515 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28769874572753906 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8326908349990845 +jet/accuracy_1_of_1‚0.35215240716934204 *jet/accuracy_1_of_3‚0.9186822175979614 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3412543535232544 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6675374507904053 7validation_average_jet_accuracy‚0.38321539759635925 _runtime‚9616.849798617 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03846890851855278 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 _step‚923 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5975438356399536 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9893202781677246 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6456054449081421 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5878808498382568 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01985609158873558 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6684925556182861 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7547430992126465 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6611143946647644 *jet/accuracy_1_of_2‚0.7788311839103699 +jet/accuracy_3_of_3‚0.04158696159720421 +jet/accuracy_4_of_4‚0.01670061983168125 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 % _timestamp‚1.7843499581556349e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0À« ü .0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7198654413223267 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20866945385932922 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029262956231832504 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7528923153877258 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5283907651901245 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7452841401100159 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7883656620979309 ,jet/accuracy_3_of_4‚0.036813318729400635 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5027045011520386 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966269135475159 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7847256660461426 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5166813731193542 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6582927107810974 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11661387979984283 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18254578113555908 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6366893649101257 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07984454929828644 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2766534388065338 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2231864482164383 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380882978439331 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0901426151394844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333134651184 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6701738834381104 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6784941554069519IÖ™|=”$öz ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +particle/specificity‚0.9006449580192566 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7668827772140503 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8267384767532349 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.098396435379982 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968291521072388 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6133191585540771 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2356867790222168 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8238095045089722 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02621140331029892 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6593807935714722 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07447560876607895 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7903681993484497 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7924433946609497 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11641217768192291 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6900812983512878 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.76036536693573 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23080816864967346 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.060833342373371124 (particle/accuracy‚0.8367165327072144 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27809709310531616 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04650595784187317 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5827711820602417 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22919043898582458 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2652595043182373 /particle/accuracy_4_of_4‚0.9539281129837036 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31657788157463074 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1512133926153183 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7333460450172424 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8763028979301453 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7051434516906738 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6751008629798889 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23650391399860382 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226568102836609 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6406713128089905 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24201788008213043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7069951891899109 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.732225775718689 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.715948760509491 particle/accuracy_1_of_1‚1 trainer/global_step‚11199 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27576276659965515 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28769874572753906 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8326908349990845 +jet/accuracy_1_of_1‚0.35215240716934204 *jet/accuracy_1_of_3‚0.9186822175979614 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3412543535232544 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6675374507904053 7validation_average_jet_accuracy‚0.38321539759635925 _runtime‚9616.849798617 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03846890851855278 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 _step‚923 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5975438356399536 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9893202781677246 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6456054449081421 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5878808498382568 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01985609158873558 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6684925556182861 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7547430992126465 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6611143946647644 *jet/accuracy_1_of_2‚0.7788311839103699 +jet/accuracy_3_of_3‚0.04158696159720421 +jet/accuracy_4_of_4‚0.01670061983168125 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 % _timestamp‚1.7843499581556349e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7198654413223267 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20866945385932922 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029262956231832504 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7528923153877258 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5283907651901245 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7452841401100159 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7883656620979309 ,jet/accuracy_3_of_4‚0.036813318729400635 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5027045011520386 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966269135475159 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7847256660461426 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5166813731193542 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6582927107810974 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11661387979984283 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18254578113555908 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6366893649101257 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07984454929828644 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2766534388065338 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2231864482164383 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380882978439331 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0901426151394844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333134651184 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6701738834381104 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6784941554069519 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2681746482849121 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9956162571907043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02637808583676815 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7446609735488892 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3250144422054291 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7164120674133301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1347973495721817 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037072885781526566 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.25982481241226196 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4993777573108673 jet/accuracy_5_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7071945667266846 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21187171339988708 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28531524538993835 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2094418704509735 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22456161677837372 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8763028979301453 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8759260773658752 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_3_of_5‚1 epoch‚27 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22433602809906006 *jet/accuracy_2_of_4‚0.7530478835105896 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28750738501548767 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8425170183181763 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675925374031067 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2766534388065338 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04125630110502243 *jet/accuracy_1_of_4‚0.9846457242965698 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8006976842880249 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.389280766248703 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6593807935714722 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7828143835067749 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5131988525390625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7809864282608032 )loss/val_total_loss‚7.647209171172678 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37834885716438293 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21089054644107819 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01985609158873558 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6578532457351685 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23521658778190613 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_2_of_2‚0.13075630366802216 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6898267269134521 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/sensitivity‚0.6029893755912781 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7116103172302246 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02901482954621315 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128›¶‚:ð•$‚(:’ “ŠìÒƒ¡†• gpu.0.gpu‚39.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.614' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.72279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 8bxò–$‚(:ç “ŠìÒòÊó— disk.rbd1.out‚ 34.5791015625 disk.sda1.out‚0 disk.dm-1.out‚ 294.7265625 disk.drbd2165.in‚0) proc.memory.percent‚16.90835952758789 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 16746.8203125( disk.nvme0c0n1.out‚27074.12255859375# disk.dm-0.out‚27074.12255859375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 294.7265625 disk.rbd1.in‚ 2.01953125$ disk./.usageGB‚588.4154281616211 disk.sda.out‚ 294.7265625 disk.sda2.out‚0( proc.memory.availableMB‚ 451.39453125 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚39.48905309735528& disk.nvme0n1.out‚27074.12255859375! proc.memory.rssMB‚ 2770.265625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.1825847837105 network.sent‚ 1.7890383e+07 network.recv‚ 2.604701e+06 disk.rbd0.out‚0$ memory_percent‚97.24490642547607 disk.dm-0.in‚ 16746.8203125 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 16746.8203125˜¡X“—$‚(:ˆ ¢ŠìÒ¥üÍ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T @ië˜$‚(:à ¢ŠìÒœÁ˜– disk.sda2.out‚0) proc.memory.percent‚16.90835952758789* disk./.usagePercent‚39.542359754157395 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2770.265625 network.sent‚ 1.7947855e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 16746.8515625# disk.dm-0.out‚27895.00732421875 disk.sda2.in‚0& disk.nvme0n1.out‚27895.00732421875' proc.memory.availableMB‚ 451.1328125 disk.dm-1.out‚295.1875 disk.sda.out‚295.1875 cpu‚26.182285411651097 disk.rbd0.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚27895.00732421875$ memory_percent‚97.24650382995605 proc.cpu.threads‚17$ disk./.usageGB‚589.2097358703613 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 34.6416015625! disk.nvme0n1.in‚ 16746.8515625 network.recv‚ 2.609362e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 16746.8515625 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚295.1875 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 LÆ2™$- _wandbruntime‚9657  _runtime‚9657‘OզȚ$ _step‚924 & lr-AdamW/pg1‚0.0001869623622128317 & lr-AdamW/pg2‚0.0001869623622128317 trainer/global_step‚11249 % _timestamp‚1.7843499986157932e+09 _runtime‚9657.313470169™*„FÍ›$Ç & lr-AdamW/pg1‚0.0001869623622128317 & lr-AdamW/pg2‚0.0001869623622128317 trainer/global_step‚11249 % _timestamp‚1.7843499986157932e+09 _runtime‚9657.313470169 _step‚924œ–Õ¾2œ$- _wandbruntime‚9657  _runtime‚96571œe3($úO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89242510 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1722428882191360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89242508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12031863783424 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104217571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376643051112448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11249 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1710525273513984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89242510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24193570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388674818192384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚9657.316369713 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11903950016512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43901258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460079 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89242508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24193833 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1722429240569856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376643032500224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1713608705421952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1722428882191360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470058 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104227548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19707691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1701722895331328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269z²'.ž$ÿO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470058 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104227548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19707691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1701722895331328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment k×$.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89242510 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1722428882191360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89242508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12031863783424 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104217571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376643051112448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11249 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1710525273513984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89242510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24193570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388674818192384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚9657.316369713 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11903950016512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43901258 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460079 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89242508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24193833 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1722429240569856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376643032500224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1713608705421952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1722428882191360â¢'S2Ÿ$- _wandbruntime‚9657  _runtime‚9657 ¬Ä ' $…N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚9657.932831062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388709268100096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19709436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376676479870976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476763 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1710672514543616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843499992389941e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376676461258752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104226623 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11904621304320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104236614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43905365 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1722577494233600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1713756227147658 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11904621304320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193476763 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89250140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11904962650624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1710672514543616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11886804105098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89250140 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1710672531582976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1710672531582976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486756 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1713755869558296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1701869423042560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43905092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24195929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24195659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104236614 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376676479870976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476763 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1710672514543616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843499992389941e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376676461258752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104226623 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11904621304320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104236614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43905365 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1722577494233600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1713756227147658 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11904621304320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193476763 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89250140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11904962650624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1710672514543616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11886804105098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89250140 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1710672531582976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1710672531582976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486756 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1713755869558296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1701869423042560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43905092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24195929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24195659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104236614 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1717887324995584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44093916 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1720984497913210 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚11299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19794941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378314470426624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1729841629174784 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1729841629174784 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1717887342034944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1709048428924928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89624110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11954645518336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24298978 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1720984855502460 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390397900545536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11936068988282 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11954304179200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306110 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12083331318272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11936409538172 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9698.434185868 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44094170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89624108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1729841987553280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89624108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390397801744896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1709048445964288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954304179200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1717887324995584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11954645518336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378314489038848 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19794938 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104672023 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1717887342034944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24299229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1729841629174784 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1729841629174784 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1717887342034944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1709048428924928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89624110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11954645518336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24298978 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1720984855502460 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390397900545536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11936068988282 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11954304179200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306110 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12083331318272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11936409538172 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9698.434185868 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44094170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89624108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1729841987553280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89624108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390397801744896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1709048445964288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954304179200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1717887324995584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11954645518336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378314489038848 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19794938 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104672023 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1717887342034944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24299229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1717887324995584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44093916 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1720984497913210 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚11299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19794941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378314470426624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378347917797376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1729990241217024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89631742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19796686 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11955658152448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1709194956636160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11955316806144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843500403571646e+09 _runtime‚9699.050929981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11955316806144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1709194939596800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11955658152448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1721132019638804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1718034566025216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89631742 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44097654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390432268551168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11937080042004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1729989882831360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚11299 _step‚930 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1729990241217024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12084369366016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1721132377228166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312815 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322808 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1718034583064576 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168–ËÊ'³$‰N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312815 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322808 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1718034583064576 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378347917797376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1729990241217024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89631742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19796686 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11955658152448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1709194956636160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11955316806144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843500403571646e+09 _runtime‚9699.050929981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11955316806144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1709194939596800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11955658152448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1721132019638804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1718034566025216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89631742 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44097654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390432268551168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11937080042004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1729989882831360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚11299 _step‚930 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1729990241217024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12084369366016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1721132377228166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¢’"©72´$- _wandbruntime‚9699  _runtime‚9699³"1U µ$‡ 0loss/FRt2/assignment_loss‚0.6327300105534547 3loss/SRqqt1/assignment_loss‚0.48757438442978307 .loss/FRt2/detection_loss‚1.468173518957984 1loss/SRqqt2/detection_loss‚0.6358559916324701 % _timestamp‚1.7843500403606582e+09 0loss/FRt1/assignment_loss‚1.4645475464365931 1loss/SRqqt1/detection_loss‚0.7618527501411736 epoch‚28 trainer/global_step‚11299 _step‚931 4loss/FBt1/assignment_loss‚5.1763008937300016e-05 2loss/FBt2/assignment_loss‚0.003991485541839866 _runtime‚9699.053966764 2loss/SRqqt2/assignment_loss‚0.4774474592641615 /loss/FRt1/detection_loss‚0.5860901212323665 /loss/FBt1/detection_loss‚0.4963202083017677 /loss/FBt2/detection_loss‚0.3972261718938912 %loss/total_loss‚7.411861411394423Žzr¶$Œ 0loss/FRt1/assignment_loss‚1.4645475464365931 1loss/SRqqt1/detection_loss‚0.7618527501411736 epoch‚28 trainer/global_step‚11299 _step‚931 4loss/FBt1/assignment_loss‚5.1763008937300016e-05 2loss/FBt2/assignment_loss‚0.003991485541839866 _runtime‚9699.053966764 2loss/SRqqt2/assignment_loss‚0.4774474592641615 /loss/FRt1/detection_loss‚0.5860901212323665 /loss/FBt1/detection_loss‚0.4963202083017677 /loss/FBt2/detection_loss‚0.3972261718938912 %loss/total_loss‚7.411861411394423 0loss/FRt2/assignment_loss‚0.6327300105534547 3loss/SRqqt1/assignment_loss‚0.48757438442978307 .loss/FRt2/detection_loss‚1.468173518957984 1loss/SRqqt2/detection_loss‚0.6358559916324701 % _timestamp‚1.7843500403606582e+09£€œ`Ñ“·$‚(:ˆ ÞŠìÒ¢³…” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.202' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ? ñ¸$‚(:æ ÞŠìÒǕÖ) proc.memory.percent‚16.90835952758789 disk.dm-0.in‚ 16760.046875 disk.sda3.out‚ 296.66796875% proc.memory.availableMB‚ 450.78125 disk.rbd0.in‚0 disk.rbd1.out‚ 34.7666015625 network.recv‚ 2.623703e+06 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 16760.046875# disk.dm-0.out‚27912.11669921875) disk./.usagePercent‚39.49084796617784 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125( disk.nvme0c0n1.out‚27912.11669921875 disk.dm-1.out‚ 296.66796875 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17$ disk./.usageGB‚588.4421730041504 disk.sda.out‚ 296.66796875 disk.sda2.out‚0& disk.nvme0n1.out‚27912.11669921875 cpu‚26.181365372855662 network.sent‚ 1.8025103e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 16760.046875 disk.rbd1.in‚ 2.01953125! proc.memory.rssMB‚ 2770.265625 disk.drbd2165.out‚0$ memory_percent‚97.24864959716797¼DTxž¹$‚(:“ íŠìÒÆ÷“ gpu.0.gpu‚43.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.844' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.968799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0| –òº$‚(:ç íŠìÒîå’–' proc.memory.availableMB‚ 451.2578125 disk.drbd2165.in‚0 disk.rbd1.out‚ 34.7666015625) proc.memory.percent‚16.90835952758789$ disk./.usageGB‚588.4421653747559 disk.sda1.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2770.265625 cpu‚26.18096025792457 network.sent‚ 1.808828e+07 disk.sda3.out‚ 297.21484375 disk.sda3.in‚ 0.1328125 network.recv‚ 2.628615e+06 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 16760.046875 disk.dm-0.in‚ 16760.046875$ memory_percent‚97.24574089050293 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 297.21484375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚27915.04736328125 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 297.21484375& disk.nvme0n1.out‚27915.04736328125 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17* disk./.usagePercent‚39.490847454162775# disk.dm-0.out‚27915.04736328125 disk.nvme0n1.in‚ 16760.046875U’ýcž»$‚(:“ üŠìÒñÖ–• gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.102' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.620400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×GhRß¼$‚(:Ô üŠìÒ”Ú™– disk.rbd0.out‚0 disk.sda.out‚297.5625$ disk.nvme0n1.out‚27920.072265625 disk.sda3.in‚ 0.1328125! proc.memory.rssMB‚ 2770.265625 cpu‚26.18070752818825 disk.sda2.in‚0 disk.nvme0n1.in‚ 16760.046875 disk.sda3.out‚297.5625' proc.memory.availableMB‚ 451.2578125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 16760.046875! disk.dm-0.out‚27920.072265625 network.recv‚ 2.631995e+06 disk.drbd2165.in‚0$ disk./.usageGB‚588.4422798156738 disk.sda1.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 34.8291015625( disk./.usagePercent‚39.4908551343888 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚27920.072265625) proc.memory.percent‚16.90835952758789 network.sent‚ 1.8093362e+07 disk.dm-1.out‚297.5625 disk.dm-0.in‚ 16760.046875 disk.sda.in‚ 0.1328125$ memory_percent‚97.24574089050293éŽ%42½$- _wandbruntime‚9739  _runtime‚9739ª¿“ʾ$Ä trainer/global_step‚11349 % _timestamp‚1.7843500805457509e+09 _runtime‚9739.239413089 _step‚932 ' lr-AdamW/pg1‚0.00018341907238166459 ' lr-AdamW/pg2‚0.00018341907238166459²zÏ¿$É trainer/global_step‚11349 % _timestamp‚1.7843500805457509e+09 _runtime‚9739.239413089 _step‚932 ' lr-AdamW/pg1‚0.00018341907238166459 ' lr-AdamW/pg2‚0.00018341907238166459¤šc´72À$- _wandbruntime‚9739  _runtime‚9739OÒ.~(Á$úO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392120854325248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195132183 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12005341020160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12004999681024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90005708 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1737254734536704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12005341020160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1737254376158208 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1737254734536704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19882188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379985926965248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12134927360000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105136452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1725249393516544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105136452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379985908353024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90005708 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1725249376477184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44285378 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12004999681024 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1737254376158208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105126475 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1725249376477184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11986685475190 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105126475 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44285114 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1728360647993718 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142162 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1728361005582968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843500805495234e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19882191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392120755524608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1716373962518528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11987026025080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0H;~eàÂ$ÿO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90005708 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1725249376477184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44285378 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12004999681024 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1737254376158208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105126475 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1725249376477184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11986685475190 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105126475 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44285114 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1728360647993718 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142162 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1728361005582968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843500805495234e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19882191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392120755524608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1716373962518528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11987026025080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392120854325248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195132183 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12005341020160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12004999681024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90005708 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1737254734536704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12005341020160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1737254376158208 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1737254734536704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19882188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379985926965248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12134927360000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105136452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1725249393516544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105136452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379985908353024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¥î-}2Ã$- _wandbruntime‚9739  _runtime‚9739w•õ 'Ä$ƒN H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1737402988200448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19883933 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44288890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11988037078914 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148867 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158860 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12006353654272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19883936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1725396617506816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392155217413632 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1728508169719312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380019355723776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚ 9739.85846339 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1728508527308674 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44289145 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24405209 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1716520473190400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90013342 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145518 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105135527 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195158860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12006012307968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12006353654272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90013342 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843500811645424e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1725396634546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380019337111552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843500811645424e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1725396634546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380019337111552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44288890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11988037078914 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148867 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158860 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12006353654272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19883936 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1725396617506816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392155217413632 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1728508169719312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380019355723776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚ 9739.85846339 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1728508527308674 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44289145 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24405209 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1716520473190400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90013342 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145518 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105135527 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195158860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12006012307968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12006353654272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90013342 BMÊÐ$Ä _step‚936 ' lr-AdamW/pg1‚0.00018165133214517572 ' lr-AdamW/pg2‚0.00018165133214517572 trainer/global_step‚11399 % _timestamp‚1.7843501215668306e+09 _runtime‚9780.260694405 Ÿ¹+ÏÑ$É ' lr-AdamW/pg1‚0.00018165133214517572 ' lr-AdamW/pg2‚0.00018165133214517572 trainer/global_step‚11399 % _timestamp‚1.7843501215668306e+09 _runtime‚9780.260694405 _step‚936¨Žà²V2Ò$- _wandbruntime‚9780  _runtime‚9780õ«î4Ó$ûO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚9780.262069622 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978214 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1744667123141632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1744667481520128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1732611427958784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freedfʧÕÍ#‚381657346279424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90387308 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1735737155663476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105580927 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90387310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105590904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44477007 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12037301962098 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44476729 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393843486527488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1723699513151488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393843602105344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381657364891648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843501215680037e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12056036521984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1735736798074226 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19969441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1744667123141632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12186140248064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195968235 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1732611444998144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12186237213696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12056036521984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24507291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1723699496112128VÏì³(Ô$€P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19969441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1744667123141632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12186140248064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195968235 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1732611444998144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12186237213696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12056036521984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24507291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1723699496112128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚9780.262069622 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978214 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1744667123141632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1744667481520128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1732611427958784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381657346279424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90387308 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1735737155663476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105580927 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90387310 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105590904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44477007 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12037301962098 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44476729 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393843486527488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1723699513151488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393843602105344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381657364891648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843501215680037e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12056036521984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1735736798074226 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438©u!ØL2Õ$- _wandbruntime‚9780  _runtime‚9780+þñº 'Ö$…N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1732758686027776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105589979 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1744815735183872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843501221855738e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24509366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12038313015820 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1732758686027776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1723846023823360 _step‚938 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12057049156096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105589979 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12187158477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1735884319799820 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚11399 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44480549 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1723846006784000 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1744815735183872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1744815376798208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195984919 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90394940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19971183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12056707809792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90394942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12038653565822 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195984919 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105599970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994912 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19971186 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381690793650176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1735884677389182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393877933515264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90394942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1zdÙÇÍ ×$ŠN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1732758686027776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105589979 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1744815735183872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843501221855738e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24509366 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12038313015820 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1732758686027776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1723846023823360 _step‚938 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12057049156096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105589979 QZp.CCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12187158477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1735884319799820 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚11399 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44480549 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1723846006784000 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1744815735183872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1744815376798208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195984919 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90394940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19971183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12056707809792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90394942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12038653565822 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195984919 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105599970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994912 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19971186 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381690793650176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1735884677389182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393877933515264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90394942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ª÷Õ”22Ø$- _wandbruntime‚9780  _runtime‚9780\“O Ù$† 0loss/FRt2/assignment_loss‚0.4670573586990412 3loss/FBt1/assignment_loss‚0.0016892722301166878 1loss/SRqqt1/detection_loss‚0.8134002019814147 0loss/FBt1/detection_loss‚0.47622404727276024 0loss/FBt2/detection_loss‚0.41541504730946455 $ _timestamp‚1.784350122189785e+09 _step‚939 0loss/FRt1/assignment_loss‚1.5117873969634525 2loss/SRqqt1/assignment_loss‚0.5201148263626529 0loss/SRqqt2/assignment_loss‚0.50250803645317 3loss/FBt2/assignment_loss‚0.0011514032166149892 /loss/FRt1/detection_loss‚0.5733724838510463 %loss/total_loss‚7.252762166438035 /loss/FRt2/detection_loss‚1.3661871059140507 epoch‚28 trainer/global_step‚11399 _runtime‚9780.883071299 1loss/SRqqt2/detection_loss‚0.60385498618425068»´Ú$‹ 1loss/SRqqt2/detection_loss‚0.6038549861842506 0loss/FRt2/assignment_loss‚0.4670573586990412 3loss/FBt1/assignment_loss‚0.0016892722301166878 1loss/SRqqt1/detection_loss‚0.8134002019814147 0loss/FBt1/detection_loss‚0.47622404727276024 0loss/FBt2/detection_loss‚0.41541504730946455 $ _timestamp‚1.784350122189785e+09 _step‚939 0loss/FRt1/assignment_loss‚1.5117873969634525 2loss/SRqqt1/assignment_loss‚0.5201148263626529 0loss/SRqqt2/assignment_loss‚0.50250803645317 3loss/FBt2/assignment_loss‚0.0011514032166149892 /loss/FRt1/detection_loss‚0.5733724838510463 %loss/total_loss‚7.252762166438035 /loss/FRt2/detection_loss‚1.3661871059140507 epoch‚28 trainer/global_step‚11399 _runtime‚9780.883071299«ä^÷:“Û$‚(:ˆ ¸‹ìÒ¿ù£” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ/€ðÜ$‚(:å ¸‹ìÒ¶Ė# disk.nvme0c0n1.in‚ 16825.7265625& disk.nvme0n1.out‚27937.57373046875 disk.rbd0.out‚0( proc.memory.availableMB‚ 450.46484375 disk.dm-0.in‚ 16825.7265625 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 299.078125$ memory_percent‚97.25058078765869( disk.nvme0c0n1.out‚27937.57373046875 disk.rbd0.in‚0! proc.memory.rssMB‚ 2770.265625! disk.nvme0n1.in‚ 16825.7265625) proc.memory.percent‚16.90835952758789$ disk./.usageGB‚588.4348602294922 disk.sda.out‚ 299.078125 disk.sda2.in‚0 disk.sda3.out‚ 299.078125 disk.drbd2165.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚39.4903571997347# disk.dm-0.out‚27937.57373046875 disk.drbd2165.out‚0 cpu‚26.179500152507725 network.sent‚ 1.8171327e+07 network.recv‚ 2.646857e+06 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 34.9541015625 disk.sda1.out‚0G°‘Ý$‚(:† Ç‹ìÒúƒÜ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÓéÞ$‚(:Þ Ç‹ìÒø¾­– disk.sda3.out‚ 299.58203125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.25100994110107* disk./.usagePercent‚39.490349007493606 disk.rbd0.in‚0) proc.memory.percent‚16.90835952758789 disk.dm-1.out‚ 299.58203125 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚16892.75 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 299.58203125 disk.sda2.out‚0 cpu‚26.179506992637695 network.recv‚ 2.651406e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚16892.75& disk.nvme0n1.out‚27941.91064453125 disk.sda3.in‚ 0.1328125 network.sent‚ 1.8234344e+07 disk.sda1.in‚0 disk.dm-0.in‚16892.75( proc.memory.availableMB‚ 450.39453125$ disk./.usageGB‚588.4347381591797 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.rbd1.out‚ 34.9541015625( disk.nvme0c0n1.out‚27941.91064453125# disk.dm-0.out‚27941.91064453125! proc.memory.rssMB‚ 2770.265625  Ö2ß$- _wandbruntime‚9821  _runtime‚9821p¨)îÊà$Ä ' lr-AdamW/pg1‚0.00017988634646949551 ' lr-AdamW/pg2‚0.00017988634646949551 trainer/global_step‚11449 % _timestamp‚1.7843501625807726e+09 _runtime‚9821.278462411 _step‚940”½BéÏá$É ' lr-AdamW/pg1‚0.00017988634646949551 ' lr-AdamW/pg2‚0.00017988634646949551 trainer/global_step‚11449 % _timestamp‚1.7843501625807726e+09 _runtime‚9821.278462411 _step‚940¬d6|Ú2â$- _wandbruntime‚9821  _runtime‚9821w-ÿb(ã$úO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1752079870125056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1752079870125056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24610500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24610763 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12088258998896 $ _timestamp‚1.784350162585736e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1739973479440384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90768910 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383328784205824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44667454 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1743112948154734 _step‚941 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106045356 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237382577152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1743113305743984 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1731025029705728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90768908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1752080228503552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106045356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395566185395200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739973496479744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12106732023808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚383328802818048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44667188 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12087918449006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12106732023808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11449 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12106390684672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106035379 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20056688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1739973496479744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‚ºe(ä$ÿO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90768908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1752080228503552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106045356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395566185395200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739973496479744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12106732023808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚383328802818048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44667188 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12087918449006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12106732023808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11449 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12106390684672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106035379 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20056688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1739973496479744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1752079870125056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1752079870125056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24610500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24610763 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12088258998896 $ _timestamp‚1.784350162585736e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1739973479440384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90768910 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383328784205824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44667454 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1743112948154734 _step‚941 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106045356 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237382577152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1743113305743984 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1731025029705728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616­Ÿ÷À2å$- _wandbruntime‚9821  _runtime‚9821 ªÙcÍæ$…N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1743260469880328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WÆW^>%IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820971 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20058433 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12089270052730 _step‚942 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12088929502728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44671243 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20058436 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1740120720470016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90776540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24612807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106044431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106054422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24612546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106054422 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1740120737509376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚9821.892731043 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196820971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383362231576576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1740120737509376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1743260827469690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106044431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12107744657920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395600639715840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1740120720470016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1752228123781632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90776542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383362212964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90776540 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843501631989768e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1752228482167296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830964 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1731171557416960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12107403311616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 trainer/global_step‚11449 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44670979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1731171540377600 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395600538825216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12238408139264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12107744657920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12238325860864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12107403311616‚Ð.'ç$ŠN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383362212964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90776540 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843501631989768e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1752228482167296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830964 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1731171557416960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12107403311616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 trainer/global_step‚11449 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44670979 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1731171540377600 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395600538825216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12238408139264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12107744657920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12238325860864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12107403311616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1743260469880328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820971 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20058433 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12089270052730 _step‚942 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12088929502728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44671243 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20058436 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1740120720470016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90776540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24612807 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106044431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106054422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24612546 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106054422 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1740120737509376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚9821.892731043 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196820971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383362231576576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1740120737509376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1743260827469690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106044431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12107744657920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395600639715840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1740120720470016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1752228123781632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90776542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169®Z¾2è$- _wandbruntime‚9821  _runtime‚9821#©…é$ˆ 1loss/SRqqt1/detection_loss‚0.8328967355795343 %loss/total_loss‚7.413218640075145 epoch‚28 $ _timestamp‚1.784350163202358e+09 _step‚943 0loss/FRt2/assignment_loss‚0.5131116152109968 2loss/SRqqt1/assignment_loss‚0.4849252610040563 4loss/FBt2/assignment_loss‚0.00020758657503415853 0loss/FBt2/detection_loss‚0.37189417136698744 0loss/FRt1/assignment_loss‚1.4649523135091438 2loss/SRqqt2/assignment_loss‚0.4956959673481275 1loss/SRqqt2/detection_loss‚0.5511739839789758 /loss/FBt1/detection_loss‚0.6521816075380359 trainer/global_step‚11449 _runtime‚9821.895673898 /loss/FRt1/detection_loss‚0.5924769328451448 .loss/FRt2/detection_loss‚1.453660058779865 4loss/FBt1/assignment_loss‚4.2406339243379014e-05Âðqê$ _runtime‚9821.895673898 /loss/FRt1/detection_loss‚0.5924769328451448 .loss/FRt2/detection_loss‚1.453660058779865 4loss/FBt1/assignment_loss‚4.2406339243379014e-05 1loss/SRqqt1/detection_loss‚0.8328967355795343 %loss/total_loss‚7.413218640075145 epoch‚28 $ _timestamp‚1.784350163202358e+09 _step‚943 0loss/FRt2/assignment_loss‚0.5131116152109968 2loss/SRqqt1/assignment_loss‚0.4849252610040563 4loss/FBt2/assignment_loss‚0.00020758657503415853 0loss/FBt2/detection_loss‚0.37189417136698744 0loss/FRt1/assignment_loss‚1.4649523135091438 2loss/SRqqt2/assignment_loss‚0.4956959673481275 1loss/SRqqt2/detection_loss‚0.5511739839789758 /loss/FBt1/detection_loss‚0.6521816075380359 trainer/global_step‚11449¯Ý|íë$‚(:’ Ö‹ìÒ’ôÆ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ô^äì$‚(:Ù Ö‹ìÒæ—º• disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚16892.75 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 299.85546875 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 35.0166015625 cpu‚26.17928949589712% disk.nvme0n1.out‚27947.0009765625 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0# memory_percent‚97.2489595413208( proc.memory.availableMB‚ 450.73046875 network.sent‚ 1.8239319e+07 network.recv‚ 2.654564e+06$ disk./.usageGB‚588.4353637695312 disk.nvme0c0n1.in‚16892.75 disk.dm-0.in‚16892.75" disk.dm-0.out‚27947.0009765625 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 299.85546875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚27947.0009765625 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 299.85546875 disk.rbd0.out‚0! proc.memory.rssMB‚ 2770.265625* disk./.usagePercent‚39.490390992729225 disk.sda2.out‚0) proc.memory.percent‚16.90835952758789ßj÷¬‘í$‚(:† å‹ìңɊ• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-·ÎÞî$‚(:Ó å‹ìÒàžö• disk.sda2.in‚0 disk.sda3.out‚ 300.05859375$ disk./.usageGB‚588.4353065490723 disk.rbd0.in‚0 disk.dm-0.out‚27950.10546875 disk.sda.out‚ 300.05859375 network.recv‚ 2.658862e+06) disk./.usagePercent‚39.49038715261621 disk.rbd0.out‚0 disk.dm-1.out‚ 300.05859375 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2770.265625 network.sent‚ 1.830227e+07 disk.sda1.out‚0 disk.dm-0.in‚16892.75 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 35.0166015625) proc.memory.percent‚16.90835952758789 proc.cpu.threads‚17% disk.nvme0c0n1.out‚27950.10546875 disk.sda1.in‚0 cpu‚26.178837052090625 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚16892.75 disk.nvme0n1.in‚16892.75 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0# disk.nvme0n1.out‚27950.10546875$ memory_percent‚97.24676609039307( proc.memory.availableMB‚ 451.08984375pi]“ï$‚(:ˆ ô‹ìÒ¨îÍ“ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç^¤,Ýð$‚(:Ò ô‹ìÒò– disk.nvme0c0n1.in‚16892.75 disk.dm-0.in‚16892.75 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 300.390625# memory_percent‚97.2468376159668& proc.memory.availableMB‚ 451.078125 network.sent‚ 1.830729e+07) disk./.usagePercent‚39.49040532915114 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 300.390625 disk.nvme0n1.in‚16892.75 disk.sda1.out‚0# disk.dm-0.out‚27955.85888671875 disk.sda.in‚ 0.1328125! proc.memory.rssMB‚ 2770.265625) proc.memory.percent‚16.90835952758789 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚ 2.662187e+06$ disk./.usageGB‚588.4355773925781 disk.rbd1.out‚ 35.0791015625 disk.rbd0.out‚0 disk.dm-1.out‚ 300.390625( disk.nvme0c0n1.out‚27955.85888671875 disk.sda2.out‚0& disk.nvme0n1.out‚27955.85888671875 disk.rbd1.in‚ 2.01953125 cpu‚26.17864101944459¢@oH2ñ$- _wandbruntime‚9862  _runtime‚9862y>Êò$Ä trainer/global_step‚11499 % _timestamp‚1.7843502033567026e+09 _runtime‚9862.050266848 _step‚944 ' lr-AdamW/pg1‚0.00017812422871635574 ' lr-AdamW/pg2‚0.00017812422871635574„X´»Ïó$É ' lr-AdamW/pg2‚0.00017812422871635574 trainer/global_step‚11499 % _timestamp‚1.7843502033567026e+09 _runtime‚9862.050266848 _step‚944 ' lr-AdamW/pg1‚0.00017812422871635574°>ÙÄ—2ô$- _wandbruntime‚9862  _runtime‚9862,(¯’õ$ûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640339 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1750489098235242 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12138534935914 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91150508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12288543949824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚945 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759492617108480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44858499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12157427525632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44858752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24714561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91150510 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24714811 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91150510 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12157427525632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1759492617108480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1738350563299328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843502033577995e+09A¦(ö$€P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759492617108480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44858499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12157427525632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44858752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24714561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91150510 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24714811 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91150510 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12157427525632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1759492617108480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1738350563299328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843502033577995e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640339 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1750489098235242 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12138534935914 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91150508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12288543949824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚945 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44862298 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12158440159744 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1747482771951616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397323331458560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106498883 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1750636619960836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1747482788990976 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106508874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1738497091010560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667016 _step‚946 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106508874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91158142 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚11499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20145683 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1747482788990976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1750636977550198 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12158098813440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91158142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24716858 _runtime‚9862.668480839 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197667016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197657023 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44862544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91158140 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385033669502976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91158140 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44862298 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12158440159744 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1747482771951616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397323331458560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106498883 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1750636619960836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1747482788990976 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106508874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1738497091010560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667016 _step‚946 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106508874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91158142 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚11499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20145683 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1747482788990976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1750636977550198 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12158098813440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91158142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24716858 _runtime‚9862.668480839 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197667016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197657023 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44862544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91158140 OADeviceStatsMoL¬ãgånitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385033669502976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91158140 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91532110 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1757865605905000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399011459640832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20231191 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1766905722470400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12339692392960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12207781688320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20231188 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106954260 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189491972712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386671678670848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12208123027456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198476391 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106954260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45048949 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24817495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386671660058624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12339780969984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1766905364091904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1754697582403584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚9903.069377898 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1754697599442944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399011352451584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91532110 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106944283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198486370 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1766905722470400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12208123027456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91532110 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1757865605905000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399011459640832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20231191 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1766905722470400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12339692392960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12207781688320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20231188 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106954260 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189491972712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386671678670848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12208123027456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198476391 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106954260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45048949 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24817495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386671660058624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12339780969984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1766905364091904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1754697582403584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚9903.069377898 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1754697599442944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399011352451584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91532110 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106944283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198486370 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269µö 2‰%- _wandbruntime‚9903  _runtime‚9903Pø¸Š%…N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91539742 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106963326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106953335 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1767053976134144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1767053617748480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399045909114880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386705107429376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12209135661568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758012770041344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91539740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1754844840472576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20232933 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1745822624604160 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843502449912276e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12340801685504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45052518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚11549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198493075 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12208794315264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1754844823433216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1767053976134144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399045801932800 _runtime‚9903.685006244 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198493075 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12208794315264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758013127630706 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1754844823433216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12190503026546 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976×L³é'‹%ŠN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45052518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚11549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198493075 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12208794315264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1754844823433216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1767053976134144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399045801932800 _runtime‚9903.685006244 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198493075 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12208794315264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758013127630706 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1754844823433216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12190503026546 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91539742 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106963326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106953335 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1767053976134144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1767053617748480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399045909114880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386705107429376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12209135661568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758012770041344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91539740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1754844840472576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20232933 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1745822624604160 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843502449912276e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12340801685504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1774318469453824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1774318111075328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚400734108004352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12390898635776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1774318111075328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45238674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1753001647525888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 $ _timestamp‚1.784350285139636e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91913710 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107408712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1762059650924544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1762059633885184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91913710 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91913708 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12390991407104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24920505 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1774318469453824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1774318111075328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚400734108004352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12390898635776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1774318111075328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45238674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1753001647525888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552¹Qœ2™%- _wandbruntime‚9944  _runtime‚9944 fÈü 'š%ƒN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107407787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400768553346560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12392007990784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1762206891954176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12259831163392 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1762206874914816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12259489817088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1774466364731904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45242713 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45242466 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20320183 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 9944.4489617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12259489817088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107417778 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12240778963452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107407787 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20320186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400768448261632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91921342 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12259831163392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12391921518080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91921340 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1774466723117568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199339120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1765389277711214 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91921340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388376545355776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1753148158197760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616p_Æ©'›%ˆN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199339120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1765389277711214 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91921340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388376545355776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1753148158197760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107407787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400768553346560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12392007990784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1762206891954176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12259831163392 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1762206874914816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12259489817088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1774466364731904 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45242713 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45242466 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20320183 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 9944.4489617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12259489817088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107417778 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12240778963452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107407787 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20320186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400768448261632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91921342 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12259831163392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12391921518080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91921340 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1774466723117568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983º˜È¨C2œ%- _wandbruntime‚9944  _runtime‚9944`3à %ƒ epoch‚28 0loss/FRt2/assignment_loss‚0.6344280153028926 2loss/SRqqt1/assignment_loss‚0.4707767680143252 2loss/SRqqt2/assignment_loss‚0.4801441421357778 /loss/FRt1/detection_loss‚0.5612499166646623 1loss/SRqqt2/detection_loss‚0.6375389745648647 trainer/global_step‚11599 3loss/FBt1/assignment_loss‚0.0009598191686835414 2loss/FBt2/assignment_loss‚0.014794745180178861 /loss/FBt2/detection_loss‚0.4378286013280475 .loss/FRt1/assignment_loss‚1.50966789528408 /loss/FRt2/detection_loss‚1.3734634045796512 1loss/SRqqt1/detection_loss‚0.8574792939930334 % _timestamp‚1.7843502857590256e+09 _runtime‚9944.452330379 _step‚955 .loss/FBt1/detection_loss‚0.445821067581001 %loss/total_loss‚7.424152643797198êyèž%ˆ _runtime‚9944.452330379 _step‚955 .loss/FBt1/detection_loss‚0.445821067581001 %loss/total_loss‚7.424152643797198 epoch‚28 0loss/FRt2/assignment_loss‚0.6344280153028926 2loss/SRqqt1/assignment_loss‚0.4707767680143252 2loss/SRqqt2/assignment_loss‚0.4801441421357778 /loss/FRt1/detection_loss‚0.5612499166646623 1loss/SRqqt2/detection_loss‚0.6375389745648647 trainer/global_step‚11599 3loss/FBt1/assignment_loss‚0.0009598191686835414 2loss/FBt2/assignment_loss‚0.014794745180178861 /loss/FBt2/detection_loss‚0.4378286013280475 .loss/FRt1/assignment_loss‚1.50966789528408 /loss/FRt2/detection_loss‚1.3734634045796512 1loss/SRqqt1/detection_loss‚0.8574792939930334 % _timestamp‚1.7843502857590256e+09»¦[Š›Ÿ%‚(: ÎŒìÒ͹ғ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òRÒó %‚(:è ÎŒìÒÍ®º– network.recv‚ 2.684569e+06 disk.sda.in‚ 0.1328125 disk.rbd0.out‚0( proc.memory.availableMB‚ 450.44921875) proc.memory.percent‚16.90835952758789! disk.nvme0n1.in‚ 17008.1484375 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0# disk.dm-0.out‚28795.91064453125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 17008.1484375 network.sent‚ 1.8453357e+07# disk./.usageGB‚588.461612701416 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.176951442464215 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 35.3291015625 proc.cpu.threads‚17 disk.sda.out‚ 302.2734375 disk.sda2.out‚0& disk.nvme0n1.out‚28795.91064453125 disk.dm-0.in‚ 17008.1484375! proc.memory.rssMB‚ 2770.265625 disk.sda1.out‚0$ memory_percent‚97.25067615509033) disk./.usagePercent‚39.49215258057233 disk.sda3.out‚ 302.2734375 disk.dm-1.out‚ 302.2734375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚28795.91064453125,Še’¡%‚(:‡ ÝŒìÒ“ýÞ“ gpu.0.gpu‚36.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚282.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ ¥ø¢%‚(:í ÝŒìÒÄÞ÷• disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 1.8518609e+07# disk.dm-0.out‚28799.06494140625 disk.rbd1.out‚ 35.3291015625 cpu‚26.180667992503587& disk.nvme0n1.out‚28799.06494140625 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 network.recv‚ 2.689521e+06* disk./.usagePercent‚39.492150532512056! disk.nvme0n1.in‚ 17008.1484375 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17$ disk./.usageGB‚588.4615821838379 disk.dm-0.in‚ 17008.1484375 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 302.74609375 disk.sda3.out‚ 302.74609375$ memory_percent‚97.30539321899414 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 17008.1484375 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.out‚ 302.74609375" proc.memory.rssMB‚ 2779.3828125* proc.memory.percent‚16.964006423950195& proc.memory.availableMB‚ 441.484375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28799.06494140625tVF#2£%- _wandbruntime‚9963  _runtime‚9963ö2Ïv¤%º{ 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751387596130371 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.512927‡‰46J=8302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6526618003845215 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31097331643104553 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093143224716187 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6207665801048279 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7590804100036621 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28650403022766113 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1304025799036026 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033138077706098557 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 particle/accuracy_2_of_5‚1 epoch‚28 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6498104929924011 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6386834383010864 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26417678594589233 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31331461668014526 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.721055269241333 +jet/accuracy_2_of_2‚0.13745348155498505 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6921294331550598 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5805199146270752 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22563119232654572 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7244141697883606 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6594842076301575 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 $ _timestamp‚1.784350304539243e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30155593156814575 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22996018826961517 *jet/accuracy_2_of_4‚0.7649338841438293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7204692959785461 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8303477168083191 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3376164436340332 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6993012428283691 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2285001426935196 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.204959899187088 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021687593311071396 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3573858141899109 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6496149897575378 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7752394080162048 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3797426223754883 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4951387941837311 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9864479303359985 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996310293674469 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5157657265663147 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8793079853057861 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6265019774436951 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6401716470718384 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.045474372804164886 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0403008870780468 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05106503143906593 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26205649971961975 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6272432804107666 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2357930839061737 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32131853699684143 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029545702040195465 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8044451475143433 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5862882733345032 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.129850372672081 7validation_average_jet_accuracy‚0.38818150758743286 _step‚956 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029767652973532677 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.937120258808136 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14756856858730316 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7532697916030884 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2503989338874817 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3564418852329254 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962859153747559 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6631680727005005 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31114840507507324 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_3‚0.9182328581809998 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8286658525466919 particle/accuracy_1_of_2‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.694703221321106 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6769338250160217 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7805113196372986 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5008658170700073 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09895770996809006 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7361960411071777 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_3_of_3‚0.04468816518783569 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6564074754714966 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955267310142517 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8793079853057861 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373785972595215 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24506668746471405 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9968254566192627 particle/accuracy_5_of_5‚0 trainer/global_step‚11599 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021687593311071396 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7954290509223938 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23748411238193512 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7"Purity/1FRt*SRqqt0FBt/event_purity‚0.23436239361763 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8183847069740295 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.253068208694458 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49755242466926575 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +particle/sensitivity‚0.6108093857765198 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31114840507507324 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8240174055099487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11284179985523224 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8269463777542114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.032786041498184204 +jet/accuracy_3_of_4‚0.05091093108057976 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751943826675415 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 *jet/accuracy_1_of_2‚0.7798442840576172 (particle/accuracy‚0.8397353291511536 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09526578336954117 *jet/accuracy_2_of_3‚0.5320165157318115 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6226171851158142 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2923920750617981 particle/accuracy_2_of_3‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24814188480377197 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3552970886230469 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3205376863479614 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7525154948234558 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04262726008892059 +particle/specificity‚0.9023439288139343 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32974761724472046 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968637824058533 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955273866653442 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5449999570846558 jet/accuracy_1_of_5‚1 _runtime‚9963.233788493 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8180649876594543 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6401716470718384 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7983158826828003 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08817311376333237 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7488473653793335 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7241055369377136 /particle/accuracy_4_of_4‚0.9380459785461426 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6760380268096924 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *loss/val_total_loss‚7.7830483950485405 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37198978662490845 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7075905203819275 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6344643235206604 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23104870319366455 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.309719055891037 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7258643507957458 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0­3ÁÅ=¥%¿{ 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24814188480377197 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3552970886230469 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3205376863479614 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7525154948234558 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04262726008892059 +particle/specificity‚0.9023439288139343 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32974761724472046 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968637824058533 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955273866653442 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5449999570846558 jet/accuracy_1_of_5‚1 _runtime‚9963.233788493 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8180649876594543 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6401716470718384 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7983158826828003 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08817311376333237 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7488473653793335 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7241055369377136 /particle/accuracy_4_of_4‚0.9380459785461426 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6760380268096924 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *loss/val_total_loss‚7.7830483950485405 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37198978662490845 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7075905203819275 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6344643235206604 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23104870319366455 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.309719055891037 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7258643507957458 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751387596130371 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6526618003845215 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31097331643104553 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093143224716187 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6207665801048279 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7590804100036621 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28650403022766113 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1304025799036026 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033138077706098557 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 particle/accuracy_2_of_5‚1 epoch‚28 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6498104929924011 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6386834383010864 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26417678594589233 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31331461668014526 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.721055269241333 +jet/accuracy_2_of_2‚0.13745348155498505 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6921294331550598 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5805199146270752 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22563119232654572 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7244141697883606 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6594842076301575 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 $ _timestamp‚1.784350304539243e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30155593156814575 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22996018826961517 *jet/accuracy_2_of_4‚0.7649338841438293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7204692959785461 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8303477168083191 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3376164436340332 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6993012428283691 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2285001426935196 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.204959899187088 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021687593311071396 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3573858141899109 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6496149897575378 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7752394080162048 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3797426223754883 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4951387941837311 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9864479303359985 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996310293674469 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5157657265663147 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8793079853057861 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6265019774436951 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6401716470718384 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.045474372804164886 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0403008870780468 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05106503143906593 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26205649971961975 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6272432804107666 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2357930839061737 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32131853699684143 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06916667520999908 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029545702040195465 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8044451475143433 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5862882733345032 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.129850372672081 7validation_average_jet_accuracy‚0.38818150758743286 _step‚956 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029767652973532677 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.937120258808136 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14756856858730316 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7532697916030884 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2503989338874817 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3564418852329254 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962859153747559 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6631680727005005 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31114840507507324 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_3‚0.9182328581809998 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8286658525466919 particle/accuracy_1_of_2‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.694703221321106 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6769338250160217 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7805113196372986 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5008658170700073 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09895770996809006 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7361960411071777 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_3_of_3‚0.04468816518783569 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6564074754714966 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955267310142517 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8793079853057861 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373785972595215 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24506668746471405 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9968254566192627 particle/accuracy_5_of_5‚0 trainer/global_step‚11599 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021687593311071396 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7954290509223938 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23748411238193512 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7"Purity/1FRt*SRqqt0FBt/event_purity‚0.23436239361763 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8183847069740295 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.253068208694458 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49755242466926575 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +particle/sensitivity‚0.6108093857765198 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31114840507507324 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8240174055099487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11284179985523224 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8269463777542114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.032786041498184204 +jet/accuracy_3_of_4‚0.05091093108057976 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751943826675415 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 *jet/accuracy_1_of_2‚0.7798442840576172 (particle/accuracy‚0.8397353291511536 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09526578336954117 *jet/accuracy_2_of_3‚0.5320165157318115 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6226171851158142 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2923920750617981 particle/accuracy_2_of_3‚1¼éXS5“¦%‚(:ˆ ìŒìÒ¬“¡” gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!~C’B§%‚(:á ìŒì҇𮖠disk.sda1.in‚0' proc.memory.availableMB‚ 437.1953125$ disk./.usageGB‚588.4616050720215 disk.rbd0.in‚0$ memory_percent‚97.33157157897949 network.sent‚ 1.8521531e+07% disk.nvme0n1.out‚28804.2197265625 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 17008.1484375* proc.memory.percent‚16.988182067871094 disk.sda.out‚ 303.265625! disk.nvme0n1.in‚ 17008.1484375 disk.dm-1.out‚ 303.265625 disk.drbd2165.in‚0 cpu‚26.181269439403568 network.recv‚ 2.69292e+06 disk.dm-0.in‚ 17008.1484375 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 35.3916015625" disk.dm-0.out‚28804.2197265625 disk.sda2.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2783.34375 proc.cpu.threads‚17 disk.sda3.out‚ 303.ØmØòª265625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125' disk.nvme0c0n1.out‚28804.2197265625) disk./.usagePercent‚39.49215206855726]]à “¨%‚(:ˆ ûŒìÒ¨üç“ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þãbÄð©%‚(:å ûŒìÒ•Ý£– disk.rbd0.out‚0$ disk./.usageGB‚588.4615898132324 disk.sda.in‚ 0.1328125 disk.sda.out‚ 303.5390625 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 303.5390625# disk.nvme0c0n1.in‚ 17068.1484375 network.recv‚ 2.697469e+06 disk.rbd1.out‚ 35.3916015625 disk.sda1.out‚0 proc.memory.rssMB‚ 2783.34375 cpu‚26.18107337405167% disk.nvme0n1.out‚28806.6572265625 proc.cpu.threads‚17 network.sent‚ 1.8579079e+07" disk.dm-0.out‚28806.6572265625! disk.nvme0n1.in‚ 17068.1484375 disk.drbd2165.out‚0$ memory_percent‚97.33030796051025 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚16.988182067871094 disk.sda1.in‚0( proc.memory.availableMB‚ 437.40234375) disk./.usagePercent‚39.49215104452713 disk.sda3.out‚ 303.5390625 disk.rbd0.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚28806.6572265625 disk.dm-0.in‚ 17068.1484375Ïímê4ª%/ _wandbruntime‚10003  _runtime‚10003yé.zË«%Å % _timestamp‚1.7843503449047692e+09 _runtime‚10003.601470319 _step‚957 ' lr-AdamW/pg1‚0.00017285621380287456 ' lr-AdamW/pg2‚0.00017285621380287456 trainer/global_step‚11649œ_\Ь%Ê ' lr-AdamW/pg1‚0.00017285621380287456 ' lr-AdamW/pg2‚0.00017285621380287456 trainer/global_step‚11649 % _timestamp‚1.7843503449047692e+09 _runtime‚10003.601470319 _step‚957½õ=Fñ4­%/ _wandbruntime‚10003  _runtime‚10003‘á1(®%úO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107923509 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327247236608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1783678516293120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327247236608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107933486 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1762235827294208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12326905897472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92415974 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402490783169536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1783678516293120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402490881970176 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92415976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1774543931416075 $ _timestamp‚1.784350344908731e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12460415346688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107933486 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92415976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1771351252017152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12326905897472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1783678157914624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1783678157914624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12460335158272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12308444671757 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390030448011264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚10003.602386809 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1771351269056512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1771351252017152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390030466623488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3+™#_(¯%ÿO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1771351269056512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45463748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200349462 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20408208 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1762235844333568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200339483 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚958 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92415976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1774543931416075 $ _timestamp‚1.784350344908731e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12460415346688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107933486 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92415976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1771351252017152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12326905897472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1783678157914624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1783678157914624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12460335158272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12308444671757 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390030448011264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚10003.602386809 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1771351269056512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1771351252017152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390030466623488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107923509 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327247236608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1783678516293120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327247236608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107933486 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1762235827294208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12326905897472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92415974 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402490783169536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1783678516293120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402490881970176 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402525340001280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1771498510086144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25057365 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92423606 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1774691453141669 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12309455725591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107932561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20409956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12461444619264 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45467318 % _timestamp‚1.7843503455240526e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1783826769956864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92423608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390063895382016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390063876769792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12328259870720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12327918524416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200356167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25057626 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1774691810731031 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20409953 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12461366535168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1762382355005440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12327918524416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚959 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92423606 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12328259870720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1771498510086144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402525340001280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1771498510086144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25057365 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92423606 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1774691453141669 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12309455725591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107932561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20409956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12461444619264 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45467318 % _timestamp‚1.7843503455240526e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1783826769956864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92423608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390063895382016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390063876769792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12328259870720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12327918524416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200356167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25057626 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1774691810731031 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20409953 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12461366535168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1762382355005440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12327918524416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚959 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92423606¿f%4³%/ _wandbruntime‚10004  _runtime‚10004¼¡W/´%ˆ 3loss/SRqqt1/assignment_loss‚0.48746637742128807 3loss/SRqqt2/assignment_loss‚0.45185657281806135 /loss/FRt1/detection_loss‚0.5982224472307526 % _timestamp‚1.7843503455278838e+09 .loss/FRt2/detection_loss‚1.485756214504363 1loss/SRqqt1/detection_loss‚0.7593580264352071 1loss/SRqqt2/detection_loss‚0.5811619149571211 _runtime‚10004.221176572 _step‚960 4loss/FBt1/assignment_loss‚0.00014867862380394653 /loss/FBt1/detection_loss‚0.5128095744533793 %loss/total_loss‚7.350566725325117 trainer/global_step‚11649 0loss/FRt1/assignment_loss‚1.4630933644852244 0loss/FRt2/assignment_loss‚0.6140573269554547 1loss/FBt2/assignment_loss‚0.01263439679123999 /loss/FBt2/detection_loss‚0.3840018306492216 epoch‚29þEµ% _step‚960 4loss/FBt1/assignment_loss‚0.00014867862380394653 /loss/FBt1/detection_loss‚0.5128095744533793 %loss/total_loss‚7.350566725325117 trainer/global_step‚11649 0loss/FRt1/assignment_loss‚1.4630933644852244 0loss/FRt2/assignment_loss‚0.6140573269554547 1loss/FBt2/assignment_loss‚0.01263439679123999 /loss/FBt2/detection_loss‚0.3840018306492216 epoch‚29 3loss/SRqqt1/assignment_loss‚0.48746637742128807 3loss/SRqqt2/assignment_loss‚0.45185657281806135 /loss/FRt1/detection_loss‚0.5982224472307526 % _timestamp‚1.7843503455278838e+09 .loss/FRt2/detection_loss‚1.485756214504363 1loss/SRqqt1/detection_loss‚0.7593580264352071 1loss/SRqqt2/detection_loss‚0.5811619149571211 _runtime‚10004.221176572À BÝ$“¶%‚(:ˆ ŠìÒ‹¦²” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s„eì·%‚(:á ŠìÒ‹õЖ network.recv‚ 2.700719e+06 disk.sda3.out‚ 303.859375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0! disk.dm-0.out‚28810.822265625( proc.memory.availableMB‚ 437.16796875* disk./.usagePercent‚39.492168197031916$ disk.nvme0n1.out‚28810.822265625 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.dm-1.out‚ 303.859375# disk.nvme0c0n1.in‚ 17068.1484375* proc.memory.percent‚16.988182067871094! disk.nvme0n1.in‚ 17068.1484375 disk.rbd0.out‚0 disk.rbd1.out‚ 35.4541015625 disk.sda1.out‚0& disk.nvme0c0n1.out‚28810.822265625 disk.sda2.in‚0$ memory_percent‚97.33173847198486 network.sent‚ 1.8584093e+07$ disk./.usageGB‚588.4618453979492 disk.sda1.in‚0 disk.sda.out‚ 303.859375 disk.rbd1.in‚ 2.01953125 disk.dm-0.in‚ 17068.1484375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 cpu‚26.180850550250366 proc.memory.rssMB‚ 2783.34375[i?ôž¸%‚(:“ ™ìÒÒ•º“ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Þä¹%‚(:Ù ™ìÒ‰òÕ– disk.dm-0.in‚ 17068.1484375 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 17068.1484375 disk.sda3.out‚ 304.390625 disk.rbd0.in‚0 disk.dm-0.out‚ 28813.2890625" disk.nvme0n1.out‚ 28813.2890625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17068.1484375$ disk.nvme0c0n1.out‚ 28813.2890625 disk.sda.out‚ 304.390625 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 disk.sda2.out‚0$ memory_percent‚97.32987880706787 network.sent‚ 1.864928e+07 network.recv‚ 2.705158e+06) disk./.usagePercent‚39.49223552701343 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.01953125 proc.memory.rssMB‚ 2783.34375* proc.memory.percent‚16.988182067871094$ disk./.usageGB‚588.4628486633301 disk.rbd1.out‚ 35.4541015625 cpu‚26.180704782508215 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 437.47265625 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 304.390625ïNþ“º%‚(:ˆ ¨ìÒ–ÿå“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚205.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Xæ"ë»%‚(:à ¨ìÒž »– disk.dm-0.in‚ 17068.1484375! disk.nvme0n1.in‚ 17068.1484375 disk.sda3.out‚ 304.71875# memory_percent‚97.3306655883789* proc.memory.percent‚16.988182067871094$ disk./.usageGB‚588.4629669189453 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚28815.98095703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda.out‚ 304.71875 disk.sda2.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚28815.98095703125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 proc.memory.rssMB‚ 2783.34375 cpu‚26.180511101647753 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125% proc.memory.availableMB‚ 437.34375 network.recv‚ 2.707387e+06* disk./.usagePercent‚39.492243463246986 disk.sda1.out‚0 disk.dm-1.out‚ 304.71875 network.sent‚ 1.8651285e+07# disk.nvme0c0n1.in‚ 17068.1484375( disk.nvme0c0n1.out‚28815.98095703125 disk.rbd0.out‚0 disk.rbd1.out‚ 35.5791015625bŠÂá4¼%/ _wandbruntime‚10044  _runtime‚10044UðòSʽ%Ä $ _timestamp‚1.784350385717759e+09 _runtime‚10044.411398175 _step‚961 ' lr-AdamW/pg1‚0.00017110669756422497 ' lr-AdamW/pg2‚0.00017110669756422497 trainer/global_step‚11699ˆ0lšÏ¾%É $ _timestamp‚1.784350385717759e+09 _runtime‚10044.411398175 _step‚961 ' lr-AdamW/pg1‚0.00017110669756422497 ' lr-AdamW/pg2‚0.00017110669756422497 trainer/global_step‚11699Á–˜ðd4¿%/ _wandbruntime‚10044  _runtime‚10044É(À%ûO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108387938 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1778713320538112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12511774896128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201175535 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45654411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20495458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201185514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12377601399296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1791091263276544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404213580645376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92797576 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1781920439085833 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1769561360887808 _runtime‚10044.412427458 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25159217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25158953 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404213679446016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391701904549888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92797574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108377961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12377601399296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12377942738432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108377961 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1791090904898048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12359061158665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0€§SüâÁ%€P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108387938 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1778713320538112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12511774896128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201175535 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45654411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20495458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201185514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12377601399296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1791091263276544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404213580645376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92797576 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1781920439085833 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1769561360887808 _runtime‚10044.412427458 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25159217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25158953 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404213679446016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391701904549888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92797574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108377961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12377601399296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12377942738432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108377961 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚11699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 J]C%”$DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1791090904898048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12359061158665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Âÿœ„V4Â%/ _wandbruntime‚10045  _runtime‚10045«¾™ 'Ã%†N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚963 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404248131829248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1782067960811539 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12359731662497 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202212 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843503863366082e+09 _runtime‚10045.030510106 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404248018355712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1791239516940288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12378614026240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11699 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108387013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92805206 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12360072212499 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92805208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201192219 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1778860561567744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45658220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1778860544528384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1778860544528384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45658497 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20497206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378614026240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1791239516940288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12512798520832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12378955372544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108387013 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108397004 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12512703659520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108397004 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1769707871559680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92805206 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45658220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1778860544528384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1778860544528384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45658497 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20497206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378614026240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1791239516940288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12512798520832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12378955372544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108387013 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108397004 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12512703659520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108397004 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1769707871559680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92805206 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚963 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404248131829248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1782067960811539 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12359731662497 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202212 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843503863366082e+09 _runtime‚10045.030510106 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404248018355712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1791239516940288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12378614026240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11699 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108387013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92805206 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12360072212499 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92805208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169ÃÅL¯M4Å%/ _wandbruntime‚10045  _runtime‚10045!àã Æ%† /loss/FRt1/assignment_loss‚1.458006195781767 2loss/FBt2/assignment_loss‚0.006306955136054337 /loss/FRt2/detection_loss‚1.3261613228549756 1loss/SRqqt1/detection_loss‚0.7736725698636078 1loss/SRqqt2/detection_loss‚0.5817534902540157 trainer/global_step‚11699 % _timestamp‚1.7843503863396792e+09 2loss/SRqqt2/assignment_loss‚0.4319956047847511 /loss/FRt1/detection_loss‚0.5688225513738998 /loss/FBt2/detection_loss‚0.2678553674996347 epoch‚29 _runtime‚10045.032976392 _step‚964 0loss/FRt2/assignment_loss‚0.5845654920672709 /loss/FBt1/detection_loss‚0.4674235298980734 %loss/total_loss‚7.024655304173311 2loss/SRqqt1/assignment_loss‚0.5521088349854836 3loss/FBt1/assignment_loss‚0.0059833896737765515#U„Ç%‹ % _timestamp‚1.7843503863396792e+09 2loss/SRqqt2/assignment_loss‚0.4319956047847511 /loss/FRt1/detection_loss‚0.5688225513738998 /loss/FBt2/detection_loss‚0.2678553674996347 epoch‚29 _runtime‚10045.032976392 _step‚964 0loss/FRt2/assignment_loss‚0.5845654920672709 /loss/FBt1/detection_loss‚0.4674235298980734 %loss/total_loss‚7.024655304173311 2loss/SRqqt1/assignment_loss‚0.5521088349854836 3loss/FBt1/assignment_loss‚0.0059833896737765515 /loss/FRt1/assignment_loss‚1.458006195781767 2loss/FBt2/assignment_loss‚0.006306955136054337 /loss/FRt2/detection_loss‚1.3261613228549756 1loss/SRqqt1/detection_loss‚0.7736725698636078 1loss/SRqqt2/detection_loss‚0.5817534902540157 trainer/global_step‚11699ÄÛûžÈ%‚(:“ ·ìҽ厔 gpu.0.gpu‚47.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇâˆâóÉ%‚(:è ·ìÒ¡ò’– network.sent‚ 1.8656361e+07" disk.nvme0n1.in‚17146.83203125 disk.sda.out‚ 305.0234375 disk.sda2.in‚0 disk.sda3.out‚ 305.0234375 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚97.33178615570068 cpu‚26.18024416063737 network.recv‚ 2.710837e+06% disk.nvme0n1.out‚28820.5869140625 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 35.5791015625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚28820.5869140625 disk.dm-0.in‚17146.83203125( proc.memory.availableMB‚ 437.16015625$ disk./.usageGB‚588.4628982543945) disk./.usagePercent‚39.49223885511137 disk.drbd2165.out‚0" disk.dm-0.out‚28820.5869140625* proc.memory.percent‚16.988182067871094 disk.rbd0.out‚0 disk.dm-1.out‚ 305.0234375 proc.memory.rssMB‚ 2783.34375 proc.cpu.threads‚17 disk.sda2.out‚0$ disk.nvme0c0n1.in‚17146.83203125 disk.sda.in‚ 0.1328125íÑ!Ê%‚(:… ÆìÒÍ€÷“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚280.815' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.163 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©1ΗíË%‚(:â ÆìÒÝÆ“–$ disk./.usageGB‚588.4628028869629 network.sent‚ 1.8719433e+07 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 network.recv‚ 2.715034e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 proc.memory.rssMB‚ 2783.34375$ disk.nvme0n1.out‚28824.056640625$ memory_percent‚97.33109474182129) disk./.usagePercent‚39.49223245492301 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚17146.83203125 cpu‚26.179976273407203 proc.cpu.threads‚17 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚28824.056640625" disk.nvme0n1.in‚17146.83203125' proc.memory.availableMB‚ 437.2734375 disk.dm-1.out‚ 305.421875$ disk.nvme0c0n1.in‚17146.83203125 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 305.421875 disk.rbd1.out‚ 35.6416015625* proc.memory.percent‚16.988182067871094 disk.drbd2165.in‚0! disk.dm-0.out‚28824.056640625 disk.sda.in‚ 0.1328125 disk.sda.out‚ 305.421875 disk.sda2.out‚0ó$5žÌ%‚(:“ ÕìÒËç“ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ƒ)ôÍ%‚(:é ÕìÒ€€Š– disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 proc.memory.rssMB‚ 2783.34375% disk.nvme0n1.out‚28828.8037109375$ disk.nvme0c0n1.in‚17146.83203125$ memory_percent‚97.33152389526367& proc.memory.availableMB‚ 437.203125* proc.memory.percent‚16.988182067871094 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.718243e+06 disk.sda.out‚ 305.98828125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚28828.8037109375 disk.sda2.out‚0 network.sent‚ 1.8724531e+07 disk.sda3.out‚ 305.98828125 disk.dm-1.out‚ 305.98828125 cpu‚26.179709617170058$ disk./.usageGB‚588.4628562927246" disk.nvme0n1.in‚17146.83203125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚28828.8037109375 disk.dm-0.in‚17146.83203125 disk.rbd1.out‚ 35.6416015625( disk./.usagePercent‚39.4922360390285à'ˆf4Î%/ _wandbruntime‚10085  _runtime‚10085-•¢ÈÏ% _step‚965 & lr-AdamW/pg1‚0.0001693606131485588 & lr-AdamW/pg2‚0.0001693606131485588 trainer/global_step‚11749 $ _timestamp‚1.784350426724746e+09 _runtime‚10085.418420617Ü×MÍÍÐ%Ç & lr-AdamW/pg1‚0.0001693606131485588 & lr-AdamW/pg2‚0.0001693606131485588 trainer/global_step‚11749 $ _timestamp‚1.784350426724746e+09 _runtime‚10085.418420617 _step‚965Åüu4Ñ%/ _wandbruntime‚10085  _runtime‚10085W%C7Ò%üO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108832413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45845351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@Devù¡ÅßãiceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25262640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93179176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20582711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108832413 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393373342476288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108842390 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93179174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843504267258909e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12428296901120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1798503651881472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93179174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1798504010259968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _runtime‚10085.419606315 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12428638240256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1786075372019712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚11749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202011587 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1786075372019712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1798503651881472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405936131454976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393373323864064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93179176 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20582708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360>Àéÿ(Ó%P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12562807590912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45845086 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12428296901120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202021566 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1786075354980352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25262378 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12409677645573 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚966 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108842390 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1798503651881472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93179174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1798504010259968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _runtime‚10085.419606315 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12428638240256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1786075372019712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚11749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202011587 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1786075372019712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1798503651881472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405936131454976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393373323864064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93179176 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20582708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108832413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45845351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25262640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93179176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20582711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108832413 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393373342476288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108842390 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93179174 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843504267258909e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12428296901120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108851456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108841465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20584456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12429650874368 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1798651905538048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1789443753302685 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108851456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843504273381803e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1798652263923712 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20584453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25264707 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93186808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1786222596009984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1786222613049344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚393406771234816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1777033422192640 _step‚967 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405970595572224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12563923130880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45848897 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚393406752622592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1789444110892047 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1798652263923712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93186808ŠÁž‹´Ö%ŠN YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1777033422192640 _step‚967 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405970595572224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12563923130880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45848897 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚393406752622592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1789444110892047 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1798652263923712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93186808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108851456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108841465 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20584456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12429650874368 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1798651905538048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1789443753302685 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108851456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843504273381803e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1798652263923712 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20584453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25264707 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93186808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1786222596009984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1786222613049344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚393406771234816Ç …¼4×%/ _wandbruntime‚10086  _runtime‚10086yz£WØ%‰ %loss/total_loss‚7.416993868501447 4loss/FBt1/assignment_loss‚2.3013899491293916e-05 2loss/FBt2/assignment_loss‚0.006937352824562039 0loss/FBt2/detection_loss‚0.34654078399401933 _step‚968 /loss/FRt2/detection_loss‚1.4820359046146123 /loss/FBt1/detection_loss‚0.6151433441308186 epoch‚29 $ _timestamp‚1.784350427341959e+09 0loss/FRt1/assignment_loss‚1.4826521368105283 0loss/FRt2/assignment_loss‚0.6007973077172917 2loss/SRqqt1/assignment_loss‚0.4760424487201929 3loss/SRqqt2/assignment_loss‚0.47499260219723033 1loss/SRqqt1/detection_loss‚0.7642448108071977 trainer/global_step‚11749 _runtime‚10086.035287386 /loss/FRt1/detection_loss‚0.5504932849456146 1loss/SRqqt2/detection_loss‚0.6170908778398876Õ Ù%Ž 0loss/FRt1/assignment_loss‚1.4826521368105283 0loss/FRt2/assignment_loss‚0.6007973077172917 2loss/SRqqt1/assignment_loss‚0.4760424487201929 3loss/SRqqt2/assignment_loss‚0.47499260219723033 1loss/SRqqt1/detection_loss‚0.7642448108071977 trainer/global_step‚11749 _runtime‚10086.035287386 /loss/FRt1/detection_loss‚0.5504932849456146 1loss/SRqqt2/detection_loss‚0.6170908778398876 %loss/total_loss‚7.416993868501447 4loss/FBt1/assignment_loss‚2.3013899491293916e-05 2loss/FBt2/assignment_loss‚0.006937352824562039 0loss/FBt2/detection_loss‚0.34654078399401933 _step‚968 /loss/FRt2/detection_loss‚1.4820359046146123 /loss/FBt1/detection_loss‚0.6151433441308186 epoch‚29 $ _timestamp‚1.784350427341959e+09È|ry‚“Ú%‚(:ˆ äìÒø®” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@þMöÛ%‚(:ë äìÒ춖 network.sent‚ 1.8729397e+07$ disk.nvme0c0n1.in‚17146.83203125 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0* proc.memory.percent‚16.988182067871094 disk.dm-0.in‚17146.83203125 disk.sda2.in‚0" disk.nvme0n1.in‚17146.83203125 disk.rbd1.out‚ 35.7041015625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚28832.22998046875 disk.sda3.out‚ 306.1484375' proc.memory.availableMB‚ 436.8203125& disk.nvme0n1.out‚28832.22998046875 disk.dm-1.out‚ 306.1484375) disk./.usagePercent‚39.49221197432028 disk.sda2.out‚0 network.recv‚ 2.721539e+06$ disk./.usageGB‚588.4624977111816 disk.sda.out‚ 306.1484375 disk.sda3.in‚ 0.1328125# disk.dm-0.out‚28832.22998046875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚97.33386039733887 proc.memory.rssMB‚ 2783.34375 cpu‚26.179490875289588 proc.cpu.threads‚17ûE6ГÜ%‚(:ˆ óìÒòÛØ“ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ŽigæÝ%‚(:Û óìÒßµó– disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚17207.92578125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 306.4453125 disk.rbd1.in‚ 2.01953125$ disk./.usageGB‚588.4624671936035 disk.rbd0.in‚0! disk.nvme0n1.out‚ 28839.421875 disk.dm-1.in‚ 0.1328125' proc.memory.availableMB‚ 437.0234375 proc.memory.rssMB‚ 2783.34375 proc.cpu.threads‚17 network.sent‚ 1.8792558e+07 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17207.92578125 disk.sda.out‚ 306.4453125" disk.nvme0n1.in‚17207.92578125 cpu‚26.1792720864772 disk.dm-1.out‚ 306.4453125 disk.drbd2165.out‚0 disk.dm-0.out‚ 28839.421875 disk.rbd1.out‚ 35.7041015625$ memory_percent‚97.33262062072754# disk.nvme0c0n1.out‚ 28839.421875 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.988182067871094* disk./.usagePercent‚39.492209926259996 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 2.726302e+06[ õ“Þ%‚(:ˆ ‚ŽìÒã²Ú” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šõ_¸òß%‚(:ç ‚ŽìÒÚÁ«– disk.sda.out‚ 306.6171875" disk.nvme0n1.in‚17207.92578125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚16.988182067871094 disk.drbd2165.in‚0$ memory_percent‚97.33390808105469 network.sent‚ 1.8797585e+07 network.recv‚ 2.72942e+06# disk.dm-0.out‚28843.77001953125 disk.sda.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚17207.92578125( disk.nvme0c0n1.out‚28843.77001953125 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 306.6171875 proc.memory.rssMB‚ 2783.34375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 306.6171875 disk.rbd1.out‚ 35.7666015625 disk.drbd2165.out‚0 disk.dm-0.in‚17207.92578125$ proc.memory.availableMB‚436.8125 disk.rbd0.in‚0) disk./.usagePercent‚39.49221735047849$ disk./.usageGB‚588.4625778198242& disk.nvme0n1.out‚28843.77001953125 cpu‚26.179034865975552ÒÐmn4à%/ _wandbruntime‚10126  _runtime‚10126ü‘«þËá%Å _step‚969 ' lr-AdamW/pg1‚0.00016761807270361537 ' lr-AdamW/pg2‚0.00016761807270361537 trainer/global_step‚11799 % _timestamp‚1.7843504675106862e+09 _runtime‚10126.206475445ÈÙ¶Ðâ%Ê ' lr-AdamW/pg1‚0.00016761807270361537 ' lr-AdamW/pg2‚0.00016761807270361537 trainer/global_step‚11799 % _timestamp‚1.7843504675106862e+09 _runtime‚10126.206475445 _step‚969É&Â_ë4ã%/ _wandbruntime‚10126  _runtime‚10126N¹1(ä%üO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93560774 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202857618 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25365617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12614038477312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202847639 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12459953582591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1793437406461952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12479333742080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12478992402944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93560774 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784212428075008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1805916757243392 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1805916398864896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109286865 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12460294132481 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1805916398864896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109286865 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109296842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20669958 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46035575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1793437406461952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25365882 _runtime‚10126.206609728 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93560776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1805916757243392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93560776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843504675114374e+09 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1796672381657599 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12479333742080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20669961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12613947803136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12478992402944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12478992402944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12459953582591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1793437406461952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12479333742080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12478992402944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93560774 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784212428075008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1805916757243392 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1805916398864896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109286865 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12460294132481 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1805916398864896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109286865 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109296842 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20669958 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46035575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1793437406461952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25365882 _runtime‚10126.206609728 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93560776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1805916757243392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93560776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843504675114374e+09 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1796672381657599 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12479333742080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20669961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12613947803136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20671703 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109295917 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12480346376192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1793584647491584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46039644 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚11799 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1793584664530944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46039374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1806065010907136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1793584664530944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407693269907456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12480346376192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25367938 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93568406 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93568408 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1796820260972555 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874316 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109305908 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12615060746240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93568408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109295917 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20671703 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109295917 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12480346376192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1793584647491584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46039644 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚11799 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1793584664530944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46039374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1806065010907136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1793584664530944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407693269907456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12480346376192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25367938 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93568406 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93568408 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1796820260972555 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874316 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269Ëœ C4é%/ _wandbruntime‚10126  _runtime‚10126hÈ5Üê%‰ 1loss/SRqqt2/detection_loss‚0.5861321247397655 /loss/FBt1/detection_loss‚0.4597850823669102 %loss/total_loss‚6.994022429375896 trainer/global_step‚11799 _runtime‚10126.826370647 /loss/FRt2/detection_loss‚1.3779258829961636 /loss/FBt2/detection_loss‚0.4303420398907217 0loss/FRt2/assignment_loss‚0.4750045011696827 4loss/FBt1/assignment_loss‚5.7624935002306446e-06 1loss/SRqqt1/detection_loss‚0.8117976327806055 0loss/FRt1/assignment_loss‚1.3923029701105145 2loss/SRqqt2/assignment_loss‚0.4282615021785051 3loss/FBt2/assignment_loss‚0.0004644573002421983 /loss/FRt1/detection_loss‚0.5676015396375261 epoch‚29 $ _timestamp‚1.784350468133061e+09 _step‚972 3loss/SRqqt1/assignment_loss‚0.46439893371175817×bZë%Ž /loss/FRt1/detection_loss‚0.5676015396375261 epoch‚29 $ _timestamp‚1.784350468133061e+09 _step‚972 3loss/SRqqt1/assignment_loss‚0.46439893371175817 1loss/SRqqt2/detection_loss‚0.5861321247397655 /loss/FBt1/detection_loss‚0.4597850823669102 %loss/total_loss‚6.994022429375896 trainer/global_step‚11799 _runtime‚10126.826370647 /loss/FRt2/detection_loss‚1.3779258829961636 /loss/FBt2/detection_loss‚0.4303420398907217 0loss/FRt2/assignment_loss‚0.4750045011696827 4loss/FBt1/assignment_loss‚5.7624935002306446e-06 1loss/SRqqt1/detection_loss‚0.8117976327806055 0loss/FRt1/assignment_loss‚1.3923029701105145 2loss/SRqqt2/assignment_loss‚0.4282615021785051 3loss/FBt2/assignment_loss‚0.0004644573002421983Ì| Û1“ì%‚(:ˆ ‘Žìқͨ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ówVÄðí%‚(:å ‘ŽìÒäòß—' disk.nvme0c0n1.out‚28849.4404296875" disk.nvme0n1.in‚17207.92578125 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 436.4296875 proc.memory.rssMB‚ 2783.34375 network.sent‚ 1.8802547e+07" disk.dm-0.out‚28849.4404296875 disk.sda.out‚ 306.921875 disk.sda3.out‚ 306.921875 network.recv‚ 2.733034e+06) disk./.usagePercent‚39.49224679134493$ disk./.usageGB‚588.4630165100098 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚28849.4404296875 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚17207.92578125 disk.rbd1.out‚ 35.7666015625$ memory_percent‚97.33624458312988* proc.memory.percent‚16.988182067871094 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 306.921875 disk.drbd2165.out‚0 disk.dm-0.in‚17207.92578125 cpu‚26.178640315754805 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0ò’ô„‘î%‚(:†  ŽìÒ“²”• gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"úšùï%‚(:î  ŽìÒâͱ–) disk./.usagePercent‚39.49224602332232 disk.sda.in‚ 0.1328125 disk.sda.out‚ 307.29296875" disk.nvme0n1.in‚17207.92578125 disk.sda1.in‚0 cpu‚26.178359913951827 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 35.8291015625$ memory_percent‚97.33507633209229 network.sent‚ 1.8865595e+07 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28853.00634765625& disk.nvme0n1.out‚28853.00634765625 disk.sda3.out‚ 307.29296875# disk./.usageGB‚588.463005065918 disk.dm-1.out‚ 307.29296875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.01953125 proc.memory.rssMB‚ 2783.34375 network.recv‚ 2.737318e+06 disk.dm-1.in‚ 0.1328125( proc.memory.availableMB‚ 436.62109375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17207.92578125 disk.sda3.in‚ 0.1328125 disk.dm-0.in‚17207.92578125# disk.dm-0.out‚28853.00634765625* proc.memory.percent‚16.988182067871094 proc.cpu.threads‚17%4‰«4ð%/ _wandbruntime‚10167  _runtime‚10167$ƒGñËñ%Å trainer/global_step‚11849 % _timestamp‚1.7843505085747926e+09 _runtime‚10167.268318813 _step‚973 ' lr-AdamW/pg1‚0.00016587918814951112 ' lr-AdamW/pg2‚0.00016587918814951112¹˜4Ðò%Ê % _timestamp‚1.7843505085747926e+09 _runtime‚10167.268318813 _step‚973 ' lr-AdamW/pg1‚0.00016587918814951112 ' lr-AdamW/pg2‚0.00016587918814951112 trainer/global_step‚11849ÍÑ&».4ó%/ _wandbruntime‚10167  _runtime‚10167^F={ ô%üO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109741317 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203693670 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46225743 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409381248907776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1804048531738107 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1800799457943552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1800799474982912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1791537978707968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409381356097024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12665137767936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93942376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12530029243904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20757211 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1800799474982912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12530029243904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93942374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109751294 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20757208 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1800799457943552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93942376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529687904768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1813329145848320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batcÁÖ¬ðâ h_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1813329145848320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396716218329088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12510570069499 % _timestamp‚1.7843505085755906e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93942374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12665049190912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ö£’(õ%P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1804048531738107 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1800799457943552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1800799474982912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1791537978707968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409381356097024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12665137767936 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93942376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12530029243904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20757211 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1800799474982912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12530029243904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93942374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109751294 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20757208 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1800799457943552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93942376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529687904768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1813329145848320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1813329145848320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396716218329088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12510570069499 % _timestamp‚1.7843505085755906e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93942374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12665049190912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109741317 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203693670 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46225743 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409381248907776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ÎŒœz4ö%/ _wandbruntime‚10167  _runtime‚10167Pl—i '÷%†N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46229469 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710368 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1800946698973184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1800946698973184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409415694011392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12530700531712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46229749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚975 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11849 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109760360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1813477757890560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1804196411053063 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1800946716012544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109750369 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12531041878016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109750369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1791684472340480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396749647087616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12511581123221 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20758956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20758953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12666065536000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12530700531712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109760360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25470793 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12511921673223 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1813477757890560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93950006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1813477399504896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843505091933963e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203700375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93950006 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚10167.887229969 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93950008 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1791684489379840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109760360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25470793 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12511921673223 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1813477757890560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93950006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1813477399504896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843505091933963e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203700375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93950006 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚10167.887229969 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93950008 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1791684489379840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46229469 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710368 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1800946698973184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1800946698973184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409415694011392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12530700531712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46229749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚975 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11849 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109760360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1813477757890560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1804196411053063 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1800946716012544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109750369 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12531041878016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109750369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1791684472340480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396749647087616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12511581123221 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20758956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freedvâ&']‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976ÏkDõ†4ù%/ _wandbruntime‚10167  _runtime‚10167ÃAÆ ú%ƒ 1loss/SRqqt1/assignment_loss‚0.434532771765846 2loss/FBt2/assignment_loss‚0.018183890610208333 1loss/SRqqt2/detection_loss‚0.5948419026550233 % _timestamp‚1.7843505091964104e+09 2loss/SRqqt2/assignment_loss‚0.3961861583909965 1loss/FBt1/assignment_loss‚0.14410210568839038 epoch‚29 0loss/FRt1/assignment_loss‚1.4263869782327225 0loss/FRt2/assignment_loss‚0.5380875001192637 1loss/SRqqt1/detection_loss‚0.8290241824245612 _runtime‚10167.889696966 _step‚976 /loss/FRt1/detection_loss‚0.5315718603967376 .loss/FRt2/detection_loss‚1.318613918013404 /loss/FBt1/detection_loss‚0.4046312144092683 /loss/FBt2/detection_loss‚0.3444182732409771 %loss/total_loss‚6.980580755947399 trainer/global_step‚11849™Û»¦û%ˆ /loss/FRt1/detection_loss‚0.5315718603967376 .loss/FRt2/detection_loss‚1.318613918013404 /loss/FBt1/detection_loss‚0.4046312144092683 /loss/FBt2/detection_loss‚0.3444182732409771 %loss/total_loss‚6.980580755947399 trainer/global_step‚11849 1loss/SRqqt1/assignment_loss‚0.434532771765846 2loss/FBt2/assignment_loss‚0.018183890610208333 1loss/SRqqt2/detection_loss‚0.5948419026550233 % _timestamp‚1.7843505091964104e+09 2loss/SRqqt2/assignment_loss‚0.3961861583909965 1loss/FBt1/assignment_loss‚0.14410210568839038 epoch‚29 0loss/FRt1/assignment_loss‚1.4263869782327225 0loss/FRt2/assignment_loss‚0.5380875001192637 1loss/SRqqt1/detection_loss‚0.8290241824245612 _runtime‚10167.889696966 _step‚976ÐAŽ6ü“ü%‚(:ˆ ¯ŽìҪ뢔 gpu.0.gpu‚85.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ÔVõý%‚(:ê ¯ŽìÒÅìÉ– network.sent‚ 1.8870616e+07 disk.rbd0.out‚0 disk.dm-1.out‚ 307.6953125 disk.sda2.in‚0 cpu‚26.177998504993464# disk.dm-0.out‚28859.31689453125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 307.6953125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.33827114105225 network.recv‚ 2.740984e+06! disk.nvme0n1.in‚ 17262.9140625& disk.nvme0n1.out‚28859.31689453125* proc.memory.percent‚16.988182067871094 proc.cpu.threads‚17$ disk./.usageGB‚588.4639282226562 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 307.6953125 disk.rbd1.out‚ 35.8291015625( proc.memory.availableMB‚ 436.09765625 proc.memory.rssMB‚ 2783.34375 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 17262.9140625 disk.dm-0.in‚ 17262.9140625( disk.nvme0c0n1.out‚28859.31689453125* disk./.usagePercent‚39.492307977145614 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125Ž«;2“þ%‚(:ˆ ¾ŽìÒö‰Ó“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚264.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05IPòÿ%‚(:ç ¾ŽìÒ±˜­– proc.cpu.threads‚17$ disk./.usageGB‚588.4637832641602 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17262.9140625" disk.dm-0.out‚28863.6044921875 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚28863.6044921875 disk.dm-0.in‚ 17262.9140625 disk.sda2.out‚0 cpu‚26.17768933804246 disk.sda3.out‚ 308.11328125 disk.rbd1.in‚ 2.01953125 network.sent‚ 1.8933852e+07) disk./.usagePercent‚39.49229824885931! disk.nvme0n1.in‚ 17262.9140625 disk.rbd1.out‚ 35.8916015625 disk.rbd0.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 436.6328125 network.recv‚ 2.745403e+06% disk.nvme0n1.out‚28863.6044921875 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 disk.dm-1.out‚ 308.11328125 disk.sda.in‚ 0.1328125 proc.memory.rssMB‚ 2783.34375 disk.rbd0.in‚0 disk.sda.out‚ 308.11328125$ memory_percent‚97.33500480651855* proc.memory.percent‚16.988182067871094ƒŒd·“€&‚(:ˆ ÍŽìҮʔ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ÎUï&‚(:ä ÍŽìÒï¶ß•) disk./.usagePercent‚39.49229133665589 disk.sda2.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 17262.9140625' disk.nvme0c0n1.out‚28871.2919921875 disk.sda.out‚ 308.7421875 disk.sda3.out‚ 308.7421875 disk.drbd2165.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚28871.2919921875* proc.memory.percent‚16.988182067871094 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 35.8916015625 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8938941e+07# disk./.usageGB‚588.463680267334 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚28871.2919921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 17262.9140625 network.recv‚ 2.748879e+06 disk.rbd0.out‚0$ memory_percent‚97.33519554138184' proc.memory.availableMB‚ 436.6015625 proc.memory.rssMB‚ 2783.34375 cpu‚26.177432032519768! disk.nvme0n1.in‚ 17262.9140625 disk.dm-1.out‚ 308.7421875,ÉV4‚&/ _wandbruntime‚10208  _runtime‚10208>é¼O˃&Å _step‚977 ' lr-AdamW/pg1‚0.00016414407117155172 ' lr-AdamW/pg2‚0.00016414407117155172 trainer/global_step‚11899 % _timestamp‚1.7843505494137738e+09 _runtime‚10208.110544876¼¨Ð„&Ê ' lr-AdamW/pg1‚0.00016414407117155172 ' lr-AdamW/pg2‚0.00016414407117155172 trainer/global_step‚11899 % _timestamp‚1.7843505494137738e+09 _runtime‚10208.110544876 _step‚977ÑÄâM4…&/ _wandbruntime‚10208  _runtime‚102080̈×(†&ûO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20844458 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12580383406592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46416340 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519743 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110195769 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1798863512301568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1811424681818615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25571882 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398387656255488 _runtime‚10208.111547249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820741892831744 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1808161526464512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110195769 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12561527106297 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12580724745728 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820741892831744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110205746 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1808161509425152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1808161509425152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94323974 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820742251210240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12580724745728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46416601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25572140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411103883595264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1808161526464512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843505494177804e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12580383406592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496ˆ²``(‡&€P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94323974 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1811425039407865 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110205746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20844461 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 trainer/global_step‚11899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12580383406592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46416340 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519743 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110195769 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1798863512301568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1811424681818615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25571882 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398387656255488 _runtime‚10208.111547249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820741892831744 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1808161526464512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110195769 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12561527106297 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12580724745728 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820741892831744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110205746 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1808161509425152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1808161509425152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94323974 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820742251210240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12580724745728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46416601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25572140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411103883595264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1808161526464512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843505494177804e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12580383406592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20844458 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25573929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411138337186816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94331606 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411138227907584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1799010005934080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1820890504873984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚11899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46420132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12562538160131 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536427 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398421066401792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20846206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12581396033536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12581737379840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25574186 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10208.728310798 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12717252172800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1820890504873984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94331608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1808308750454784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12562197610129 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 $ _timestamp‚1.784350550034444e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94331608 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46420132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12562538160131 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536427 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398421066401792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20846206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12581396033536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12581737379840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25574186 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10208.728310798 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12717252172800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1820890504873984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94331608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1808308750454784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12562197610129 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 $ _timestamp‚1.784350550034444e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94331608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25573929 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411138337186816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94331606 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411138227907584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1799010005934080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1820890504873984Óbdµ{4‹&/ _wandbruntime‚10208  _runtime‚10208ôÆu2 Œ&‡ epoch‚29 3loss/SRqqt2/assignment_loss‚0.49056757005881596 2loss/FBt2/assignment_loss‚0.008637051397725535 /loss/FRt1/detection_loss‚0.5545732252995199 .loss/FRt2/detection_loss‚1.409759743689168 /loss/FBt1/detection_loss‚0.4474812563823755 % _timestamp‚1.7843505500380266e+09 0loss/FRt2/assignment_loss‚0.6771793603033259 0loss/SRqqt2/detection_loss‚0.633000549185759 0loss/FBt2/detection_loss‚0.40815268103182106 /loss/FRt1/assignment_loss‚1.535980893442264 2loss/SRqqt1/assignment_loss‚0.4743582017516667 %loss/total_loss‚7.428143144207789 trainer/global_step‚11899 _runtime‚10208.731344315 _step‚980 4loss/FBt1/assignment_loss‚0.00013473140532023066 1loss/SRqqt1/detection_loss‚0.7883178802600276[8ž&Œ /loss/FRt1/assignment_loss‚1.535980893442264 2loss/SRqqt1/assignment_loss‚0.4743582017516667 %loss/total_loss‚7.428143144207789 trainer/global_step‚11899 _runtime‚10208.731344315 _step‚980 4loss/FBt1/assignment_loss‚0.00013473140532023066 1loss/SRqqt1/detection_loss‚0.7883178802600276 epoch‚29 3loss/SRqqt2/assignment_loss‚0.49056757005881596 2loss/FBt2/assignment_loss‚0.008637051397725535 /loss/FRt1/detection_loss‚0.5545732252995199 .loss/FRt2/detection_loss‚1.409759743689168 /loss/FBt1/detection_loss‚0.4474812563823755 % _timestamp‚1.7843505500380266e+09 0loss/FRt2/assignment_loss‚0.6771793603033259 0loss/SRqqt2/detection_loss‚0.633000549185759 0loss/FBt2/detection_loss‚0.40815268103182106Ô3ú|¹žŽ&‚(:“ ÜŽìÒ‹¼Å” gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¼Xî&‚(:ã ÜŽìÒµª‰˜) disk./.usagePercent‚39.49229338471616$ disk./.usageGB‚588.4637107849121 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.rbd1.out‚ 35.9541015625 disk.sda1.out‚0# disk.dm-0.out‚28875.58544921875 network.sent‚ 1.8956855e+07 network.recv‚ 2.752078e+06 disk.drbd2165.in‚0 disk.sda.out‚ 308.96875 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 17262.9140625( disk.nvme0c0n1.out‚28875.58544921875 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 308.96875 disk.sda2.in‚0( proc.memory.availableMB‚ 436.12890625 proc.memory.rssMB‚ 2783.34375* proc.memory.percent‚16.988182067871094& disk.nvme0n1.out‚28875.58544921875 cpu‚26.177159835382806 disk.drbd2165.out‚0 disk.dm-0.in‚ 17262.9140625 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 308.96875! disk.nvme0n1.in‚ 17262.9140625$ memory_percent‚97.33808040618896 proc.cpu.threads‚17ëÓÙC“&‚(:ˆ ëŽìÒöèÌ• gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.099' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ì‹zõ‘&‚(:ê ëŽìÒêÛ— disk.sda1.out‚0 disk.dm-1.out‚ 309.51953125 disk.sda.out‚ 309.51953125 proc.memory.rssMB‚ 2783.34375* proc.memory.percent‚16.988182067871094 cpu‚26.177003413247405( proc.memory.availableMB‚ 436.79296875 network.recv‚ 2.756605e+06* disk./.usagePercent‚39.492303113002464 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.33402729034424 disk.sda3.out‚ 309.51953125 disk.dm-1.in‚ 0.1328125" disk.nvme0c0n1.in‚ 17337.796875 disk.sda2.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.out‚ 35.9541015625 disk.drbd2165.out‚0# disk.dm-0.out‚28882.04736328125& disk.nvme0n1.out‚28882.04736328125 disk.rbd0.in‚0 network.sent‚ 1.9006992e+07 disk.nvme0n1.in‚ 17337.796875 disk.rbd1.in‚ 2.01953125( disk.nvme0c0n1.out‚28882.04736328125 disk.dm-0.in‚ 17337.796875 proc.cpu.threads‚17$ disk./.usageGB‚588.4638557434082 disk.sda1.in‚0Ê•r™‘’&‚(:† úŽìÒŠÅè” gpu.0.gpu‚58.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øä“&‚(:Ù úŽìң節$ disk./.usageGB‚588.4640197753906 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0 disk.dm-1.out‚ 309.71875 disk.drbd2165.in‚0 network.sent‚ 1.9012184e+07 disk.dm-0.in‚ 17337.796875 disk.nvme0n1.in‚ 17337.796875$ disk.nvme0n1.out‚28886.642578125 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 309.71875* proc.memory.percent‚16.988182067871094 cpu‚26.176700436992565 disk.sda1.in‚0& disk.nvme0c0n1.out‚28886.642578125' proc.memory.availableMB‚ 436.0234375 network.recv‚ 2.759692e+06 disk.sda2.out‚0 disk.sda3.out‚ 309.71875 disk.rbd1.out‚ 36.0166015625! disk.dm-0.out‚28886.642578125 disk.sda.in‚ 0.1328125$ memory_percent‚97.33872413635254) disk./.usagePercent‚39.49231412132643 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125" disk.nvme0c0n1.in‚ 17337.796875 proc.cpu.threads‚17 proc.memory.rssMB‚ 2783.34375rÓ8B4”&/ _wandbruntime‚10249  _runtime‚102493ÿ>Ë•&Å ' lr-AdamW/pg1‚0.00016241283321305827 ' lr-AdamW/pg2‚0.00016241283321305827 trainer/global_step‚11949 % _timestamp‚1.7843505904147153e+09 _runtime‚10249.108269966 _step‚9817NçÅЖ&Ê ' lr-AdamW/pg1‚0.00016241283321305827 ' lr-AdamW/pg2‚0.00016241283321305827 trainer/global_step‚11949 % _timestamp‚1.7843505904147153e+09 _runtime‚10249.108269966 _step‚981Õ†Á Ç4—&/ _wandbruntime‚10249  _runtime‚10249´Ã:(˜&ûO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365774 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94705576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12631078908416 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1828154998193664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚11949 % _timestamp‚1.7843505904154797e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1815523560906752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20931711 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25674977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚10249.109135981 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1815523577946112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20931708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828154639815168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828154998193664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12631078908416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400059094181888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12767447951360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110650221 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1815523560906752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94705574 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46606685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412826523521024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12611803043315 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94705574 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1818801189488373 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94705576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110650221 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12631420247552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110660198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412826622321664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25675229 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12631420247552 _step‚982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1806189028855808.>˜j›™&€P WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828154998193664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12631078908416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400059094181888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12767447951360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110650221 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1815523560906752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94705574 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46606685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDro k!eviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412826523521024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12611803043315 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94705574 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1818801189488373 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94705576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110650221 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12631420247552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110660198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412826622321664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25675229 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12631420247552 _step‚982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1806189028855808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365774 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94705576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12631078908416 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1828154998193664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚11949 % _timestamp‚1.7843505904154797e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1815523560906752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20931711 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25674977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚10249.109135981 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1815523577946112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20931708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828154639815168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101Ösot4š&/ _wandbruntime‚10249  _runtime‚10249Ã0”ä '›&„N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1815670801936384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12768480025088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25677333 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚983 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110659273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12612814097037 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25677078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12632091535360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94713208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1828302893471744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12632432881664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400092522940416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11949 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12632432881664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205372479 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1806335539527680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205382472 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46610531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110659273 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94713206 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12632091535360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1818948711214079 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1815670801936384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚10249.727809538¢SÂu'œ&‰N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12612814097037 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25677078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12632091535360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94713208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1828302893471744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12632432881664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400092522940416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11949 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12632432881664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205372479 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1806335539527680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205382472 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46610531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110659273 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94713206 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12632091535360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1818948711214079 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1815670801936384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚10249.727809538 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1815670801936384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12768480025088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25677333 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚983 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110659273×<£Do4&/ _wandbruntime‚10249  _runtime‚10249Ôˆž&ˆ 1loss/SRqqt2/detection_loss‚0.6187175235271557 /loss/FBt1/detection_loss‚0.5392712664026289 %loss/total_loss‚7.670413655291995 _runtime‚10249.730278479 0loss/FRt2/assignment_loss‚0.7440330223332073 2loss/SRqqt1/assignment_loss‚0.6090775457525999 3loss/FBt1/assignment_loss‚0.0017356901808457136 2loss/FBt2/assignment_loss‚0.017636695587887042 /loss/FRt1/detection_loss‚0.5299299877194731 /loss/FRt2/detection_loss‚1.4949663893412595 _step‚984 0loss/FRt1/assignment_loss‚1.5382759910715778 % _timestamp‚1.7843505910369623e+09 2loss/SRqqt2/assignment_loss‚0.4418022465433477 1loss/SRqqt1/detection_loss‚0.7498245613856448 0loss/FBt2/detection_loss‚0.38514273544636746 epoch‚29 trainer/global_step‚11949ÇO‘ÑŸ& /loss/FBt1/detection_loss‚0.5392712664026289 %loss/total_loss‚7.670413655291995 _runtime‚10249.730278479 0loss/FRt2/assignment_loss‚0.7440330223332073 2loss/SRqqt1/assignment_loss‚0.6090775457525999 3loss/FBt1/assignment_loss‚0.0017356901808457136 2loss/FBt2/assignment_loss‚0.017636695587887042 /loss/FRt1/detection_loss‚0.5299299877194731 /loss/FRt2/detection_loss‚1.4949663893412595 _step‚984 0loss/FRt1/assignment_loss‚1.5382759910715778 % _timestamp‚1.7843505910369623e+09 2loss/SRqqt2/assignment_loss‚0.4418022465433477 1loss/SRqqt1/detection_loss‚0.7498245613856448 0loss/FBt2/detection_loss‚0.38514273544636746 epoch‚29 trainer/global_step‚11949 1loss/SRqqt2/detection_loss‚0.6187175235271557ØT’ &‚(:‡ ‰ìÒÌß»” gpu.0.gpu‚19.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É Ü^ä¡&‚(:Ù ‰ìÒËÈš— disk.dm-1.out‚ 310.0234375 disk.dm-0.in‚ 17337.796875 disk.sda.out‚ 310.0234375 disk.rbd1.out‚ 36.0166015625 proc.memory.rssMB‚ 2783.34375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 17337.796875( proc.memory.availableMB‚ 436.26953125 disk.dm-0.out‚ 28893.9296875 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 310.0234375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0" disk.nvme0n1.out‚ 28893.9296875 proc.cpu.threads‚17 network.sent‚ 1.9017061e+07 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 17337.796875$ disk.nvme0c0n1.out‚ 28893.9296875 cpu‚26.17651116029036 disk.sda1.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.988182067871094 network.recv‚ 2.763004e+06 disk.rbd1.in‚ 2.01953125# memory_percent‚97.3372220993042* disk./.usagePercent‚39.492332041853835$ disk./.usageGB‚588.4642868041992 disk.dm-1.in‚ 0.1328125×XK“¢&‚(:ˆ ˜ìÒ·Ø“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆötN£&‚(:ä ˜ìÒä¡€– disk.rbd1.in‚ 2.01953125 cpu‚26.17621194271785$ disk./.usageGB‚588.4639625549316 disk.rbd1.out‚ 36.0791015625 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 310.61328125 network.sent‚ 1.9080236e+07) disk./.usagePercent‚39.49231028121342 disk.sda.out‚ 310.61328125 disk.nvme0n1.in‚ 17337.796875' proc.memory.availableMB‚ 436.2734375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 17337.796875* proc.memory.percent‚16.988182067871094 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 17337.796875% disk.nvme0n1.out‚28898.0810546875 network.recv‚ fÆuR2.767494e+06 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125$ memory_percent‚97.33719825744629 proc.memory.rssMB‚ 2783.34375 disk.rbd0.out‚0 disk.dm-1.out‚ 310.61328125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚28898.0810546875" disk.dm-0.out‚28898.0810546875 disk.sda2.in‚0 disk.sda3.in‚ 0.13281250³›¸ž¤&‚(:“ §ìÒ׮ē gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ã°ð¥&‚(:å §ìÒ˜À÷•$ disk./.usageGB‚588.4639472961426 disk.sda3.out‚ 310.8515625 disk.rbd1.out‚ 36.0791015625 proc.cpu.threads‚17 network.recv‚ 2.770831e+06 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125* proc.memory.percent‚16.988182067871094 cpu‚26.17588347994751 disk.sda.out‚ 310.8515625" disk.nvme0n1.in‚17395.83984375$ disk.nvme0n1.out‚28903.423828125! disk.dm-0.out‚28903.423828125 network.sent‚ 1.9085095e+07 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125& disk.nvme0c0n1.out‚28903.423828125 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 436.22265625 disk.sda.in‚ 0.1328125$ memory_percent‚97.33750820159912 proc.memory.rssMB‚ 2783.34375 disk.dm-0.in‚17395.83984375 disk.dm-1.out‚ 310.8515625 disk.rbd1.in‚ 2.01953125) disk./.usagePercent‚39.49230925718329 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17395.83984375 disk.sda2.in‚0 disk.drbd2165.out‚0Ù­¸·4¦&/ _wandbruntime‚10290  _runtime‚10290< ]eɧ&à & lr-AdamW/pg2‚0.0001606855854682098 trainer/global_step‚11999 % _timestamp‚1.7843506314497693e+09 _runtime‚10290.143223965 _step‚985 & lr-AdamW/pg1‚0.0001606855854682098›æŠ©Î¨&È trainer/global_step‚11999 % _timestamp‚1.7843506314497693e+09 _runtime‚10290.143223965 _step‚985 & lr-AdamW/pg1‚0.0001606855854682098 & lr-AdamW/pg2‚0.0001606855854682098Ù>þÑK4©&/ _wandbruntime‚10290  _runtime‚10290ÆÖ·(ª&ûO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12662419530223 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95087176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚11999 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12682115749376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21018961 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1826176981979631 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1822885629427712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12818913713152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12682115749376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1822885612388352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46797575 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12662760080113 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46797837 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401730532108288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843506314505634e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201826 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401730513496064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111104673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414549445821440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1813514562449408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111114650 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1826177339568881 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚986 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1835567386798592 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12681774410240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1835567386798592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1835567745177088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95087174 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21018958]ñ‰-(«&€P TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95087176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚11999 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12682115749376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21018961 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1826176981979631 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1822885629427712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12818913713152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12682115749376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1822885612388352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46797575 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12662760080113 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46797837 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401730532108288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843506314505634e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201826 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401730513496064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111104673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414549445821440 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1813514562449408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111114650 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1826177339568881 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚986 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1835567386798592 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12681774410240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1835567386798592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1835567745177088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95087174 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21018958 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12662419530223ÚcÏö4¬&/ _wandbruntime‚10290  _runtime‚10290wcâ^™&­&†N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414583900764672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414583791485440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1823032853417984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46801618 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12683128383488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218524 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823032853417984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10290.761860697 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1835715640455168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1835715998840832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218524 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12663430583945 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95094808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12663771133947 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25780912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813661090160640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1835715998840832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401763960866816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12819849230848 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1813661073121280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46801356 trainer/global_step‚11999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95094808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111123716 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12682787037184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1826324861294587 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206208531 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12819939897856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/r{²å)sequested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0å þò'®&‹N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401763960866816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12819849230848 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1813661073121280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46801356 trainer/global_step‚11999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95094808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111123716 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12682787037184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1826324861294587 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206208531 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12819939897856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414583900764672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414583791485440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1823032853417984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46801618 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12683128383488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218524 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823032853417984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10290.761860697 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1835715640455168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1835715998840832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218524 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12663430583945 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95094808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12663771133947 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25780912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813661090160640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1835715998840832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360Û—ÝÄš4¯&/ _wandbruntime‚10290  _runtime‚10290ù‘4 °&… 0loss/FRt1/assignment_loss‚1.5344521140887495 2loss/SRqqt2/assignment_loss‚0.4130989130511933 2loss/FBt2/assignment_loss‚0.024430265145954543 /loss/FRt1/detection_loss‚0.5366840698038242 .loss/FRt2/detection_loss‚1.257667237282787 2loss/SRqqt1/assignment_loss‚0.4719085467805256 3loss/FBt1/assignment_loss‚7.198498596078049e-05 0loss/FBt2/detection_loss‚0.46230571589180997 %loss/total_loss‚7.191550831274066 % _timestamp‚1.7843506320714066e+09 _runtime‚ 10290.7646974 0loss/FRt2/assignment_loss‚0.5418941419611695 1loss/SRqqt1/detection_loss‚0.8095568486059079 1loss/SRqqt2/detection_loss‚0.5638684633230581 epoch‚29 trainer/global_step‚11999 _step‚988 /loss/FBt1/detection_loss‚0.5756125303531259Fòò±&Š 1loss/SRqqt2/detection_loss‚0.5638684633230581 epoch‚29 trainer/global_step‚11999 _step‚988 /loss/FBt1/detection_loss‚0.5756125303531259 0loss/FRt1/assignment_loss‚1.5344521140887495 2loss/SRqqt2/assignment_loss‚0.4130989130511933 2loss/FBt2/assignment_loss‚0.024430265145954543 /loss/FRt1/detection_loss‚0.5366840698038242 .loss/FRt2/detection_loss‚1.257667237282787 2loss/SRqqt1/assignment_loss‚0.4719085467805256 3loss/FBt1/assignment_loss‚7.198498596078049e-05 0loss/FBt2/detection_loss‚0.46230571589180997 %loss/total_loss‚7.191550831274066 % _timestamp‚1.7843506320714066e+09 _runtime‚ 10290.7646974 0loss/FRt2/assignment_loss‚0.5418941419611695 1loss/SRqqt1/detection_loss‚0.8095568486059079ܳ u‘²&‚(:† ¶ìÒþ……• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ˆ(Ló³&‚(:è ¶ìÒ¹Íú• disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 36.1416015625 disk.sda1.in‚0 disk.dm-0.in‚17395.83984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125% proc.memory.availableMB‚ 428.40625" proc.memory.rssMB‚ 2791.1328125* proc.memory.percent‚17.035722732543945 proc.cpu.threads‚17 network.recv‚ 2.774048e+06$ disk./.usageGB‚588.4640197753906 disk.sda3.in‚ 0.1328125 network.sent‚ 1.9090071e+07) disk./.usagePercent‚39.49231412132643 disk.sda3.out‚ 311.171875( disk.nvme0c0n1.out‚28906.83740234375$ memory_percent‚97.38521575927734& disk.nvme0n1.out‚28906.83740234375 disk.drbd2165.in‚0 disk.sda.out‚ 311.171875" disk.nvme0n1.in‚17395.83984375 disk.dm-1.out‚ 311.171875# disk.dm-0.out‚28906.83740234375 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚17395.83984375 cpu‚26.179256931117948c¬Ë4´&/ _wandbruntime‚10309  _runtime‚10309²î¦«­=µ&§{ 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2307015359401703 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3208693861961365 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 'particle/f_score‚0.6240847706794739 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6693803668022156 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8889902234077454 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028716543689370155 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21847647428512573 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9924398064613342 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20369082689285278 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49844127893447876 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971954345703125 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.6173685789108276 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6957619190216064 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7906462550163269 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2846739590167999 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7167014479637146 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6538094878196716 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020867053419351578 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23159955441951752 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7088319063186646 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.109283946454525 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20580008625984192 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6validation_average_jet_accuracy‚0.3768180012702942 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8685941100120544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27072614431381226 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25230222940444946 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9881864190101624 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2624014914035797 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026278577744960785 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23091177642345428 *jet/accuracy_1_of_2‚0.7789416313171387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843506506646569e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6975966095924377 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.736859917640686 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08815503865480423 +jet/accuracy_3_of_3‚0.04658283665776253 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6336399912834167 _runtime‚10309.358833802 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6516979932785034 ,jet/accuracy_3_of_4‚0.046922553330659866 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 jet/accuracy_5_of_5‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07716397196054459 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2137051373720169 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964054822921753 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.227263405919075 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02581894025206566 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5631769299507141 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5201533436775208 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5052107572555542 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6135358810424805 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605892419815063 *jet/accuracy_2_of_2‚0.1318083256483078 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975752830505371 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.31883782148361206 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1870659589767456 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6157292127609253 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6227843761444092 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 _step‚989 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975752830505371 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03411312773823738 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09086354821920395 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 *jet/accuracy_2_of_3‚0.5239977836608887 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6174639463424683 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2814074456691742 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6816969513893127 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8863345384597778 )loss/val_total_loss‚7.638813287928806 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_4_of_4‚0.9586439728736877 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8510582447052002 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.226848766207695 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_5‚0 epoch‚29 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2347089648246765 (particle/accuracy‚0.8402894139289856 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23016135394573212 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2832692265510559 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9524383544921875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21840417385101318 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03688333183526993 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27199453115463257 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950646758079529 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7840441465377808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3698148727416992 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37798988819122314 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8863345384597778 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975563883781433 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04608864337205887 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21862615644931793 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4964892864227295 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27072614431381226 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957783222198486 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.663347065448761 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.809732973575592 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6414710283279419 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7556790113449097 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7125154137611389 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.70089191198349 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09963487088680267 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972967505455017 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6568303108215332 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328324317932129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6410027146339417 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1268477886915207 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3086262047290802 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )jet/accuracy_2_of_4‚0.784274697303772 +jet/accuracy_4_of_4‚0.01628045178949833 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10834785550832748 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7468012571334839 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9207659363746643 +particle/specificity‚0.9012575745582581 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965649843215942 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6414710283279419 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5575350522994995 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996768712997437 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9917007684707642 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 *jet/accuracy_1_of_1‚0.3425678014755249 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33534833788871765 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7987924218177795 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7368545532226562 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980986714363098 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.273570716381073 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04032529145479202 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6814068555831909 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7482138276100159 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30233579874038696 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8745465278625488 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7787497639656067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7966156005859375 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020867053419351578 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8830310106277466 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚11999 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7620607018470764 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959343671798706 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8466552495956421 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6286576986312866 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02759549953043461 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8753¹>‚¶&¬{ 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2846739590167999 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7167014479637146 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6538094878196716 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020867053419351578 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23159955441951752 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7088319063186646 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.109283946454525 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20580008625984192 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6validation_average_jet_accuracy‚0.3768180012702942 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8685941100120544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27072614431381226 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25230222940444946 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9881864190101624 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2624014914035797 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026278577744960785 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23091177642345428 *jet/accuracy_1_of_2‚0.7789416313171387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843506506646569e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6975966095924377 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.736859917640686 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08815503865480423 +jet/accuracy_3_of_3‚0.04658283665776253 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6336399912834167 _runtime‚10309.358833802 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6516979932785034 ,jet/accuracy_3_of_4‚0.046922553330659866 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 jet/accuracy_5_of_5‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07716397196054459 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2137051373720169 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964054822921753 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.227263405919075 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02581894025206566 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5631769299507141 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5201533436775208 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5052107572555542 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6135358810424805 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605892419815063 *jet/accuracy_2_of_2‚0.1318083256483078 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9931805133819wP¼&0/58 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975752830505371 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.31883782148361206 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1870659589767456 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6157292127609253 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6227843761444092 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 _step‚989 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975752830505371 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03411312773823738 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09086354821920395 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 *jet/accuracy_2_of_3‚0.5239977836608887 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6174639463424683 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2814074456691742 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6816969513893127 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8863345384597778 )loss/val_total_loss‚7.638813287928806 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_4_of_4‚0.9586439728736877 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8510582447052002 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.226848766207695 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_5‚0 epoch‚29 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2347089648246765 (particle/accuracy‚0.8402894139289856 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23016135394573212 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2832692265510559 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9524383544921875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21840417385101318 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03688333183526993 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27199453115463257 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950646758079529 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7840441465377808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3698148727416992 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37798988819122314 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8863345384597778 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975563883781433 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04608864337205887 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21862615644931793 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4964892864227295 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27072614431381226 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957783222198486 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.663347065448761 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.809732973575592 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6414710283279419 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7556790113449097 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7125154137611389 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.70089191198349 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09963487088680267 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972967505455017 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6568303108215332 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328324317932129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6410027146339417 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1268477886915207 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3086262047290802 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )jet/accuracy_2_of_4‚0.784274697303772 +jet/accuracy_4_of_4‚0.01628045178949833 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10834785550832748 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7468012571334839 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9207659363746643 +particle/specificity‚0.9012575745582581 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965649843215942 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6414710283279419 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5575350522994995 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996768712997437 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9917007684707642 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 *jet/accuracy_1_of_1‚0.3425678014755249 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33534833788871765 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7987924218177795 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7368545532226562 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980986714363098 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.273570716381073 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04032529145479202 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6814068555831909 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7482138276100159 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30233579874038696 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8745465278625488 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7787497639656067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7966156005859375 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020867053419351578 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8830310106277466 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚11999 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7620607018470764 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959343671798706 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8466552495956421 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6286576986312866 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02759549953043461 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2307015359401703 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3208693861961365 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 'particle/f_score‚0.6240847706794739 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6693803668022156 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8889902234077454 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028716543689370155 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21847647428512573 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9924398064613342 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20369082689285278 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49844127893447876 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7971954345703125 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.6173685789108276 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6957619190216064 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7906462550163269Ý„m3cž·&‚(:“ ÅìÒ·¹ß” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚230.817' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.163399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03þÞºí¸&‚(:â ÅìÒ©ªî• disk.sda.in‚ 0.1328125' proc.memory.availableMB‚ 424.4921875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 311.53125 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚28911.7978515625$ disk./.usageGB‚588.4640655517578 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚17395.83984375* disk./.usagePercent‚39.492317193416845 disk.sda2.in‚0" disk.nvme0n1.in‚17395.83984375 disk.rbd1.out‚ 36.1416015625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚28911.7978515625 disk.dm-0.in‚17395.83984375 disk.sda3.in‚ 0.1328125$ memory_percent‚97.40910530090332 cpu‚26.18030211916422 proc.cpu.threads‚17 network.sent‚ 1.9153111e+07 network.recv‚ 2.7784e+06* proc.memory.percent‚17.061185836791992 disk.rbd1.in‚ 2.01953125 disk.sda.out‚ 311.53125 disk.dm-1.out‚ 311.53125" proc.memory.rssMB‚ 2795.3046875% disk.nvme0n1.out‚28911.7978515625 disk.sda1.out‚0¤8/ú’¹&‚(:‡ ÔìÒÂêÞ“ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨L;ûº&‚(:ð ÔìÒÊúœ– disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚28914.27001953125 disk.sda.out‚ 311.95703125 disk.sda3.out‚ 311.95703125 disk.dm-1.out‚ 311.95703125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚17395.83984375 disk.dm-0.in‚17395.83984375 disk.sda2.out‚0" disk.nvme0n1.in‚17395.83984375& disk.nvme0n1.out‚28914.27001953125$ memory_percent‚97.40931987762451( disk./.usagePercent‚39.4923056730778 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125( proc.memory.availableMB‚ 424.45703125 proc.cpu.threads‚17 network.sent‚ 1.9210464e+07$ disk./.usageGB‚588.4638938903809 disk.sda1.in‚0 cpu‚26.180159829596843 disk.sda3.in‚ 0.1328125( disk.nvme0c0n1.out‚28914.27001953125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 36.2666015625* proc.memory.percent‚17.061185836791992 network.recv‚ 2.782499e+06 disk.sda2.in‚0" proc.memory.rssMB‚ 2795.3046875€4»&/ _wandbruntime‚10349  _runtime‚10349ØÕ˼&Å % _timestamp‚1.7843506910947511e+09 _runtime‚10349.788325748 _step‚990 ' lr-AdamW/pg1‚0.00015896243887490125 ' lr-AdamW/pg2‚0.00015896243887490125 trainer/global_step‚12049òVñüн&Ê ' lr-AdamW/pg2‚0.00015896243887490125 trainer/global_step‚12049 % _timestamp‚1.7843506910947511e+09 _runtime‚10349.788325748 _step‚990 ' lr-AdamW/pg1‚0.00015896243887490125ÞÝ2/[4¾&/ _wandbruntime‚10349  _runtime‚10349™qT8(¿&ûO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12730755742360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403417882134528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12750544456704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12888071179264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95589440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1832177230520320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25914201 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403417863522304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843506910957508e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218887 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1844927433637888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1844927433637888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12750544456704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207228866 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1822748776296448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1832177247559680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1822748759257088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1844927792016384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12750203117568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12750203117568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1835479872588698 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111629447 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47023186 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21108728 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111629447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1835479514999448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12731096292250 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚10349.789470088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111639424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1832177247559680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21108731 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12730755742360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403417882134528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12750544456704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12888071179264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95589440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1832177230520320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25914201 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403417863522304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843506910957508e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218887 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1844927433637888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1844927433637888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12750544456704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207228866 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1822748776296448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1832177247559680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1822748759257088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1844927792016384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12750203117568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12750203117568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceS'¦lžŸ tatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1835479872588698 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111629447 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47023186 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21108728 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111629447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1835479514999448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12731096292250 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚10349.789470088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111639424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1832177247559680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21108731 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1832324471549952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47026727 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207235571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12751215744512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207235571 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416340484693504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1845076045680128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403451310893056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1845076045680128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1832324488589312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1832324488589312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95597072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25916254 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843506917118797e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12732107346084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95597072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12751557090816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21110473 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416340375414272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111648490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111648490 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111638499 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403451292280832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12731766796082 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21110476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95597074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95597074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822895286968320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10350.405632208 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25916516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207245564 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111638499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚12049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47026992y¹úõ'Å&‹N trainer/global_step‚12049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47026992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1832324471549952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1835627036725042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12751215744512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1832324471549952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47026727 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207235571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12751215744512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207235571 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416340484693504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1845076045680128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403451310893056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1845076045680128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1832324488589312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1832324488589312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95597072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25916254 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843506917118797e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12732107346084 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95597072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12751557090816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21110473 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416340375414272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111648490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111648490 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111638499 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403451292280832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12731766796082 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21110476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95597074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95597074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822895286968320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚10350.405632208 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25916516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207245564 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111638499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1à,Óf4Æ&/ _wandbruntime‚10350  _runtime‚10350¶){øÇ&ˆ /loss/FBt1/detection_loss‚0.3733988833458473 epoch‚30 0loss/FRt1/assignment_loss‚1.4439109676498392 1loss/SRqqt2/assignment_loss‚0.448241282717078 4loss/FBt1/assignment_loss‚0.00010008715492152273 3loss/FBt2/assignment_loss‚0.0033720825129129966 /loss/FRt1/detection_loss‚0.5768804752516064 1loss/SRqqt1/detection_loss‚0.7968356500941891 2loss/SRqqt1/assignment_loss‚0.5459285208953014 /loss/FRt2/detection_loss‚1.3728153332823096 1loss/SRqqt2/detection_loss‚0.5905410287899723 % _timestamp‚1.7843506917146008e+09 _runtime‚10350.407901932 _step‚993 0loss/FRt2/assignment_loss‚0.6749976277351379 trainer/global_step‚12049 /loss/FBt2/detection_loss‚0.3066857133094616 %loss/total_loss‚7.133707652738577û\ùýÈ& 0loss/FRt2/assignment_loss‚0.6749976277351379 trainer/global_step‚12049 /loss/FBt2/detection_loss‚0.3066857133094616 %loss/total_loss‚7.133707652738577 /loss/FBt1/detection_loss‚0.3733988833458473 epoch‚30 0loss/FRt1/assignment_loss‚1.4439109676498392 1loss/SRqqt2/assignment_loss‚0.448241282717078 4loss/FBt1/assignment_loss‚0.00010008715492152273 3loss/FBt2/assignment_loss‚0.0033720825129129966 /loss/FRt1/detection_loss‚0.5768804752516064 1loss/SRqqt1/detection_loss‚0.7968356500941891 2loss/SRqqt1/assignment_loss‚0.5459285208953014 /loss/FRt2/detection_loss‚1.3728153332823096 1loss/SRqqt2/detection_loss‚0.5905410287899723 % _timestamp‚1.7843506917146008e+09 _runtime‚10350.407901932 _step‚993áÐ’EžÉ&‚(:“ òìÒ¥‘ý“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÆ/ñÊ&‚(:æ òìÒ¾¹ – disk.sda.in‚ 0.1328125 disk.sda2.in‚0% disk.nvme0n1.out‚28922.1787109375 disk.rbd1.out‚ 36.3291015625 disk.dm-1.out‚ 312.53515625 network.recv‚ 2.790457e+06 disk.sda.out‚ 312.53515625 disk.rbd0.in‚0$ memory_percent‚97.40867614746094$ proc.memory.availableMB‚424.5625 cpu‚26.179615642245423 network.sent‚ 1.926783e+07# disk.nvme0c0n1.in‚ 17456.0546875 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 312.53515625 disk.sda1.in‚0" disk.dm-0.out‚28922.1787109375 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.49231821744698$ disk./.usageGB‚588.4640808105469! disk.nvme0n1.in‚ 17456.0546875 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0" proc.memory.rssMB‚ 2795.3046875' disk.nvme0c0n1.out‚28922.1787109375 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0* proc.memory.percent‚17.061185836791992 disk.dm-0.in‚ 17456.0546875 disk.sda2.out‚0 disk.drbd2165.out‚0›ë-“Ë&‚(:ˆ ìÒûÔ›” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÐEaðÌ&‚(:å ìÒôÃ’–* proc.memory.percent‚17.061209678649902 disk.dm-1.out‚ 313.03125 disk.rbd1.in‚ 2.01953125# disk.dm-0.out‚28926.01025390625 disk.sda.out‚ 313.03125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 36.3291015625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 17456.0546875! disk.nvme0n1.in‚ 17456.0546875# proc.memory.rssMB‚ 2795.30859375 proc.cpu.threads‚17 network.recv‚ 2.794535e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚28926.01025390625$ disk./.usageGB‚588.4639854431152 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 17456.0546875' proc.memory.availableMB‚ 424.5546875 cpu‚26.179501928656233 network.sent‚ 1.9320191e+07 disk.sda1.in‚0& disk.nvme0n1.out‚28926.01025390625 disk.sda3.out‚ 313.03125$ memory_percent‚97.40872383117676) disk./.usagePercent‚39.49231181725863 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 ¸A34Í&/ _wandbruntime‚10390  _runtime‚10390&sì&ÊÎ&Ä ' lr-AdamW/pg2‚0.00015724350410761847 trainer/global_step‚12099 $ _timestamp‚1.784350731941995e+09 _runtime‚10390.635508358 _step‚994 ' lr-AdamW/pg1‚0.00015724350410761847Ç+w»ÏÏ&É ' lr-AdamW/pg1‚0.00015724350410761847 ' lr-AdamW/pg2‚0.00015724350410761847 trainer/global_step‚12099 $ _timestamp‚1.784350731941995e+09 _runtime‚10390.635508358 _step‚994âr Е4Ð&/ _wandbruntime‚10390  _runtime‚10390½F§L± Ñ&úO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95971042 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112093876 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1852340538999808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418028832668160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1852340180621312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26018819 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12939512607232 =!‰ÂwO6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1842855665079956 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1852340180621312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12800898619392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405089320060928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1839539299041280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95971040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12801239958528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95971042 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47214541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12781712779158 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21195981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26018563 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112083899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 $ _timestamp‚1.784350731942726e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112093876 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚405089301448704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47214800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208064918 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1842856022669206 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12800898619392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1830074292850688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1839539282001920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1839539282001920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781372229268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95971040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12939432418816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418028733867520h†(Ò&ÿO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21195981 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26018563 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112083899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 $ _timestamp‚1.784350731942726e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112093876 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚405089301448704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47214800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208064918 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1842856022669206 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12800898619392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1830074292850688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1839539282001920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1839539282001920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781372229268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95971040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12939432418816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418028733867520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95971042 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112093876 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1852340538999808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418028832668160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1852340180621312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26018819 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12939512607232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1842855665079956 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1852340180621312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12800898619392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405089320060928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1839539299041280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95971040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12801239958528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95971042 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47214541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12781712779158㎪?ø4Ó&/ _wandbruntime‚10391  _runtime‚10391~Õ 'Ô&†N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12802252592640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112102942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782723832992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843507325567968e+09 _runtime‚10391.250534297 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95978672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112092951 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26020654 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12940541536256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12802252592640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801911246336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47218625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚12099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21197726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418063181076480 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112092951 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12801911246336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47218377 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚996 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208071623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26020899 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418063290355712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1852488792663552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405122748819456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1843003544394912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95978674 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248TúnEIÕ&‹N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208071623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26020899 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418063290355712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1852488792663552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405122748819456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1843003544394912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95978674 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12802252592640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112102942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782723832992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843507325567968e+09 _runtime‚10391.250534297 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95978672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112092951 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26020654 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12940541536256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12802252592640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801911246336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47218625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚12099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21197726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418063181076480 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112092951 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12801911246336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47218377 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚996 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0äÓ›¬4Ö&/ _wandbruntime‚10391  _runtime‚10391j%Ô ×&‡ 1loss/SRqqt1/detection_loss‚0.8349965575291772 _step‚997 .loss/FBt1/detection_loss‚0.567744091352714 %loss/total_loss‚7.563682466932873 % _timestamp‚1.7843507325730603e+09 _runtime‚10391.266430993 0loss/FRt1/assignment_loss‚1.4190404348068164 3loss/FBt1/assignment_loss‚7.731367741062708e-06 .loss/FRt1/detection_loss‚0.581642954409737 /loss/FBt2/detection_loss‚0.4612306142802588 epoch‚30 trainer/global_step‚12099 0loss/FRt2/assignment_loss‚0.6825170698063225 4loss/FBt2/assignment_loss‚0.00037486683453040866 .loss/FRt2/detection_loss‚1.464549122564063 1loss/SRqqt2/detection_loss‚0.6725111403760065 2loss/SRqqt1/assignment_loss‚0.4168096083554122 3loss/SRqqt2/assignment_loss‚0.46225827525009316WÈžëØ&Œ 0loss/FRt1/assignment_loss‚1.4190404348068164 3loss/FBt1/assignment_loss‚7.731367741062708e-06 .loss/FRt1/detection_loss‚0.581642954409737 /loss/FBt2/detection_loss‚0.4612306142802588 epoch‚30 trainer/global_step‚12099 0loss/FRt2/assignment_loss‚0.6825170698063225 4loss/FBt2/assignment_loss‚0.00037486683453040866 .loss/FRt2/detection_loss‚1.464549122564063 1loss/SRqqt2/detection_loss‚0.6725111403760065 2loss/SRqqt1/assignment_loss‚0.4168096083554122 3loss/SRqqt2/assignment_loss‚0.46225827525009316 1loss/SRqqt1/detection_loss‚0.8349965575291772 _step‚997 .loss/FBt1/detection_loss‚0.567744091352714 %loss/total_loss‚7.563682466932873 % _timestamp‚1.7843507325730603e+09 _runtime‚10391.266430993å‰ÕdÄžÙ&‚(:“ ìÒØÜ¥“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì Ò´ùÚ&‚(:î ìÒëÿÞ—$ disk.nvme0c0n1.in‚17456.69921875 cpu‚26.179258858796263 disk.dm-0.in‚17456.69921875 disk.sda3.out‚ 313.59765625 disk.rbd0.in‚0$ memory_percent‚96.55694961547852" proc.memory.rssMB‚ 2636.9765625 network.recv‚ 2.797728e+06$ disk./.usageGB‚588.4643211364746( disk.nvme0c0n1.out‚28929.20166015625 disk.sda2.out‚0& disk.nvme0n1.out‚28929.20166015625# disk.dm-0.out‚28929.20166015625 disk.sda.out‚ 313.59765625 disk.sda1.out‚0 disk.dm-1.out‚ 313.59765625 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 1.9325181e+07 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd1.out‚ 36.3916015625 disk.drbd2165.out‚0" disk.nvme0n1.in‚17456.69921875& proc.memory.availableMB‚ 564.109375) disk./.usagePercent‚39.49233434592164 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125) proc.memory.percent‚16.09482765197754 disk.sda3.in‚ 0.1328125 $/“Û&‚(:ˆ ŸìÒ٨œ gpu.0.gpu‚79.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿%ÃõÜ&‚(:ê ŸìÒïóÕ•( proc.memory.availableMB‚ 564.35546875 network.recv‚ 2.802404e+06 disk.sda2.out‚0 disk.sda3.out‚ 313.7890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 313.7890625 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚17456.69921875# proc.memory.rssMB‚ 2636.98828125 disk.sda1.out‚0 disk.sda.in‚ 0.1328125 network.sent‚ 1.9388495e+07) disk./.usagePercent‚39.49231847345452 disk.drbd2165.out‚0" disk.dm-0.out‚28936.1982421875 cpu‚26.17913178890825 proc.cpu.threads‚17 disk.dm-0.in‚17456.69921875 disk.dm-1.in‚ 0.1328125 disk.sda2.in‚0" disk.nvme0n1.in‚17456.69921875 disk.rbd1.out‚ 36.3916015625) proc.memory.percent‚16.09489917755127 disk.rbd1.in‚ 2.01953125$ memory_percent‚96.55544757843018$ disk./.usageGB‚588.4640846252441% disk.nvme0n1.out‚28936.1982421875 disk.sda1.in‚0' disk.nvme0c0n1.out‚28936.1982421875 disk.sda.out‚ 313.7890625DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283228 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406760739375104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12990878621184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1846901333483520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96352642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1859753285983232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900970 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419751636608512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12832329266066 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96352640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1850232172749714 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1837399843483648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843507730046062e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚10431.698313274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1846901350522880 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1837399826444288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1846901333483520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12990785849856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26122031 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96352642 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47405259 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859752927604736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26122301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96352640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112538351 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1846901350522880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419751525224960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12831988716176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12851935460352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12851935460352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112548328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406760757987328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250¼ÙË×(ä&P ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1837399826444288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1846901333483520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12990785849856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26122031 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96352642 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47405259 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859752927604736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26122301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96352640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112538351 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1846901350522880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419751525224960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12831988716176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12851935460352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12851935460352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112548328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406760757987328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12851594121216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1859753285983232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47405532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1850231815160464 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1859752927604736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208890991 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _step‚999 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12851594121216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112548328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283228 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406760739375104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12990878621184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1846901333483520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96352642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1859753285983232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900970 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419751636608512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12832329266066 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96352640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1850232172749714 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1837399843483648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843507730046062e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚10431.698313274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1846901350522880çRÑ+ë4å&/ _wandbruntime‚10432  _runtime‚10432É+ø{æ&‡N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96360274 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208907675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1859901181261312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitorᇰµ’#.on_train_batch_end/active.large_pool.allocated‚96360274 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12852606748160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12852948094464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917668 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47408993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1850379336886058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96360272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1859901539646976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1859901539646976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26124279 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419785976734720 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1850379694475420 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12832999769898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12852948094464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26124020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112547403 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96360272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12991895073792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21284973 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112547403 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12852606748160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406794168133632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112557394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1847048591552512 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26124020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112547403 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96360272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12991895073792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21284973 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112547403 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12852606748160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406794168133632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112557394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1847048591552512 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12852948094464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917668 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47408993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1850379336886058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591èÒcNX4è&/ _wandbruntime‚10432  _runtime‚10432’N´‰ é&† 1loss/SRqqt2/detection_loss‚0.6259613363490535 /loss/FRt1/assignment_loss‚1.501266827214251 /loss/FRt1/detection_loss‚0.5472871468217596 0loss/SRqqt1/detection_loss‚0.897625562135079 0loss/FBt2/detection_loss‚0.46355238547744193 trainer/global_step‚12149 % _timestamp‚1.7843507736254132e+09 _step‚1001 0loss/FRt2/assignment_loss‚0.6893208891222482 2loss/SRqqt1/assignment_loss‚0.5692280070006274 %loss/total_loss‚7.538466056552158 epoch‚30 2loss/FBt1/assignment_loss‚0.011326145745330021 .loss/FBt1/detection_loss‚0.306007766753043 _runtime‚10432.318721111 2loss/SRqqt2/assignment_loss‚0.4899306924134732 3loss/FBt2/assignment_loss‚0.0013073929989685928 /loss/FRt2/detection_loss‚1.4356519045208833¡oPê&‹ 2loss/FBt1/assignment_loss‚0.011326145745330021 .loss/FBt1/detection_loss‚0.306007766753043 _runtime‚10432.318721111 2loss/SRqqt2/assignment_loss‚0.4899306924134732 3loss/FBt2/assignment_loss‚0.0013073929989685928 /loss/FRt2/detection_loss‚1.4356519045208833 1loss/SRqqt2/detection_loss‚0.6259613363490535 /loss/FRt1/assignment_loss‚1.501266827214251 /loss/FRt1/detection_loss‚0.5472871468217596 0loss/SRqqt1/detection_loss‚0.897625562135079 0loss/FBt2/detection_loss‚0.46355238547744193 trainer/global_step‚12149 % _timestamp‚1.7843507736254132e+09 _step‚1001 0loss/FRt2/assignment_loss‚0.6893208891222482 2loss/SRqqt1/assignment_loss‚0.5692280070006274 %loss/total_loss‚7.538466056552158 epoch‚30éCÅ×8ë&‚(:’ ½ìÒÃíÛ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.224' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.84479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÐ üì&‚(:ñ ½ìÒÐâö—) disk./.usagePercent‚39.49231847345452 disk.rbd0.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚28947.07373046875 disk.rbd1.out‚ 36.4541015625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.01953125( disk.nvme0c0n1.out‚28947.07373046875$ disk.nvme0c0n1.in‚17516.65234375 disk.sda3.in‚ 0.1328125$ memory_percent‚96.55606746673584) proc.memory.percent‚16.09489917755127 cpu‚26.178726147286017 network.sent‚ 1.9397607e+07 network.recv‚ 2.808335e+06 disk.dm-0.in‚17516.65234375 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 564.25390625 disk.sda3.out‚ 314.33984375# proc.memory.rssMB‚ 2636.98828125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 314.33984375# disk.dm-0.out‚28947.07373046875 disk.sda.out‚ 314.33984375 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚17516.65234375$ disk./.usageGB‚588.4640846252441 disk.sda1.in‚0ã0ZÔí&‚(:’ ÌìÒ¸´ª” gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚252.389' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.47780000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¿}ñùî&‚(:î ÌìÒ˜•——) proc.memory.percent‚16.09489917755127 cpu‚26.178474298286602 network.sent‚ 1.9460736e+07 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 314.87890625 disk.rbd0.in‚0 disk.dm-0.in‚17516.65234375 disk.sda.out‚ 314.87890625" disk.nvme0n1.in‚17516.65234375 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚588.4641151428223 disk.drbd2165.out‚0" disk.dm-0.out‚28949.6044921875 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 564.52734375 network.recv‚ 2.812881e+06$ disk.nvme0c0n1.in‚17516.65234375' disk.nvme0c0n1.out‚28949.6044921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 314.87890625 disk.rbd1.out‚ 36.5166015625# proc.memory.rssMB‚ 2636.98828125$ memory_percent‚96.55439853668213) disk./.usagePercent‚39.49232052151479 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚28949.6044921875 disk.sda1.in‚0 disk.rbd1.in‚ 2.01953125—Z“ï&‚(:ˆ ÛìÒ…çÝ“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ßÇÈîð&‚(:ã ÛìÒÅð“— disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 315.1953125$ disk./.usageGB‚588.4642372131348 disk.sda1.in‚0) proc.memory.percent‚16.09489917755127 disk.sda1.out‚0 disk.sda.in‚ 0.1328125# disk.nvme0n1.out‚28953.28515625& proc.memory.availableMB‚ 564.265625 network.sent‚ 1.9466048e+07) disk./.usagePercent‚39.49232871375589 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚17516.65234375 disk.sda2.out‚0 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚28953.28515625 disk.sda.out‚ 315.1953125" disk.nvme0n1.in‚17516.65234375 disk.sda3.out‚ 315.1953125 disk.rbd1.out‚ 36.5166015625# proc.memory.rssMB‚ 2636.98828125 cpu‚26.178171787165088 disk.sda2.in‚0$ memory_percent‚96.55599594116211 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.in‚ 2.01953125 disk.dm-0.in‚17516.65234375 disk.dm-0.out‚28953.28515625 disk.sda3.in‚ 0.1328125 network.recv‚ 2.816541e+06 disk.rbd0.in‚0cD X4ñ&/ _wandbruntime‚10472  _runtime‚10472[ìËò&Å $ _timestamp‚1.784350813808769e+09 _runtime‚10472.502403024 _step‚1002 ' lr-AdamW/pg1‚0.00015381871138939337 ' lr-AdamW/pg2‚0.00015381871138939337 trainer/global_step‚12199³å°õÐó&Ê trainer/global_step‚12199 $ _timestamp‚1.784350813808769e+09 _runtime‚10472.502403024 _step‚1002 ' lr-AdamW/pg1‚0.00015381871138939337 ' lr-AdamW/pg2‚0.00015381871138939337ê>u, 4ô&/ _wandbruntime‚10472  _runtime‚10472-ÐKèõ&üO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408432195913728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1857608322830222 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21370478 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112992803 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96734240 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96734242 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26225416 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12902289623040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12902630962176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1867166032966656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737022 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47596155 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12902630962176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408432177301504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113002780 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13042051123200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1867165674588160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47595894 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784350813812547e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1003 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112992803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854263384965120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12882605203084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113002780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854263384965120 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1867165674588160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854263402004480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26225674 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1844725377077248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1844725360037888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12902289623040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421474228424704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26225416 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12902289623040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12902630962176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1867166032966656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737022 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47596155 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12902630962176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408432177301504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113002780 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13042051123200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1867165674588160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47595894 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784350813812547e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1003 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112992803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854263384965120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12882605203084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113002780 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854263384965120 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1867165674588160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854263402004480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26225674 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1844725377077248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1844725360037888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12902289623040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421474228424704 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13043158196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12903302249984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784350814429738e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209753720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854410643034112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1854410643034112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113011846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12903643596288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26227440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47599663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1004 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12903643596288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96741872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113001855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21372223 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1867314286630400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12883616256806 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421508782868992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854410625994752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96741874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12883956806808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10473.123710768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1857755486966566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47599917 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372226 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96741874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12903302249984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209743727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421508679881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Q;à|$ù&‹N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12883616256806 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421508782868992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854410625994752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96741874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12883956806808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10473.123710768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1857755486966566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47599917 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372226 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96741874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12903302249984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209743727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421508679881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209743727 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1867314286630400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13043158196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12903302249984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784350814429738e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209753720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854410643034112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1854410643034112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113011846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12903643596288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26227440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47599663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1004 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12903643596288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96741872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113001855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.¸$•current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21372223 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1867314286630400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224ìBf¦Ò4ú&/ _wandbruntime‚10473  _runtime‚10473²jÀ– û&… 0loss/FRt2/assignment_loss‚0.6696904916917125 2loss/SRqqt2/assignment_loss‚0.5101692442458202 .loss/FRt1/detection_loss‚0.530710506805769 /loss/FRt2/detection_loss‚1.6130084074113897 0loss/SRqqt2/detection_loss‚0.611878054736827 trainer/global_step‚12199 % _timestamp‚1.7843508144325774e+09 _step‚1005 2loss/SRqqt1/assignment_loss‚0.5736550963820987 3loss/FBt1/assignment_loss‚0.0009525283201805947 1loss/FBt2/assignment_loss‚0.03889130871926409 /loss/FBt2/detection_loss‚0.3878120410967515 %loss/total_loss‚7.873500078555081 _runtime‚10473.125873771 0loss/FRt1/assignment_loss‚1.4900474627542912 1loss/SRqqt1/detection_loss‚0.8242588439366755 epoch‚30 /loss/FBt1/detection_loss‚0.6224260924543005¨Ñc5ü&Š 2loss/SRqqt1/assignment_loss‚0.5736550963820987 3loss/FBt1/assignment_loss‚0.0009525283201805947 1loss/FBt2/assignment_loss‚0.03889130871926409 /loss/FBt2/detection_loss‚0.3878120410967515 %loss/total_loss‚7.873500078555081 _runtime‚10473.125873771 0loss/FRt1/assignment_loss‚1.4900474627542912 1loss/SRqqt1/detection_loss‚0.8242588439366755 epoch‚30 /loss/FBt1/detection_loss‚0.6224260924543005 0loss/FRt2/assignment_loss‚0.6696904916917125 2loss/SRqqt2/assignment_loss‚0.5101692442458202 .loss/FRt1/detection_loss‚0.530710506805769 /loss/FRt2/detection_loss‚1.6130084074113897 0loss/SRqqt2/detection_loss‚0.611878054736827 trainer/global_step‚12199 % _timestamp‚1.7843508144325774e+09 _step‚1005íR'm<žý&‚(:“ êìÒæÌ×” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Üw-ôþ&‚(:é êìÒÄÙŽ– disk.rbd0.in‚0 disk.sda.in‚ 0.1328125" proc.memory.availableMB‚564.25 network.recv‚ 2.819633e+06# disk.dm-0.out‚28956.42724609375 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125$ disk./.usageGB‚588.4642524719238 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚17577.39453125& disk.nvme0n1.out‚28956.42724609375 disk.sda3.out‚ 315.40234375 cpu‚26.1779705219878 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚17577.39453125# proc.memory.rssMB‚ 2636.98828125 disk.dm-1.out‚ 315.40234375) proc.memory.percent‚16.09489917755127 proc.cpu.threads‚17 disk.rbd1.out‚ 36.5791015625 network.sent‚ 1.9470981e+07) disk./.usagePercent‚39.49232973778602 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28956.42724609375 disk.sda.out‚ 315.40234375 disk.sda2.in‚0$ memory_percent‚96.55609130859375 disk.sda2.out‚0" disk.nvme0n1.in‚17577.39453125¿é9’ÿ&‚(:‡ ùìÒ­ÄГ gpu.0.gpu‚34.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚198.802' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.7604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rõ ø€'‚(:í ùìÒåÓ¸– disk.sda3.in‚ 0.1328125 network.sent‚ 1.9533997e+07 disk.sda2.in‚0( disk.nvme0c0n1.out‚28960.96630859375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2636.98828125 proc.cpu.threads‚17$ memory_percent‚96.55382633209229( proc.memory.availableMB‚ 564.62109375) disk./.usagePercent‚39.49227751224904 disk.rbd1.out‚ 36.5791015625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0# disk.dm-0.out‚28960.96630859375 disk.rbd1.in‚ 2.01953125 disk.dm-1.out‚ 315.6953125 cpu‚26.17776277438973 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17577.39453125) proc.memory.percent‚16.09489917755127 network.recv‚ 2.824053e+06$ disk./.usageGB‚588.4634742736816 disk.sda.out‚ 315.6953125" disk.nvme0n1.in‚17577.39453125 disk.dm-0.in‚17577.39453125& disk.nvme0n1.out‚28960.96630859375 disk.sda3.out‚ 315.6953125ž–1ã4'/ _wandbruntime‚10513  _runtime‚10513E¾8Ì‚'Æ ' lr-AdamW/pg1‚0.00015211307340648757 ' lr-AdamW/pg2‚0.00015211307340648757 trainer/global_step‚12249 % _timestamp‚1.7843508548357694e+09 _runtime‚10513.532427175 _step‚10068Xjaу'Ë ' lr-AdamW/pg2‚0.00015211307340648757 trainer/global_step‚12249 % _timestamp‚1.7843508548357694e+09 _runtime‚10513.532427175 _step‚1006 ' lr-AdamW/pg1‚0.00015211307340648757îç·4„'/ _wandbruntime‚10513  _runtime‚10513óBJY(…'üO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12953326464000 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1861625453486080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚12249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47786549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21457728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1861625436446720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874578421571584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97115840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113447255 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1852050910670848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12952985124864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47786799 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410103615227904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12953326464000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410103633840128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874578421571584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784350854839717e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97115842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1864984472910730 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874578779950080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1007 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113457232 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423196921765376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968Cj÷(†'P G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21457731 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26328821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210563095 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚423197028954624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13093306537472 _runtime‚10513.533376788 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210573074 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12952985124864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12933221689992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26329068 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1861625453486080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚12249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47786549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21457728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1861625436446720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874578421571584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97115840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113447255 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1852050910670848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12952985124864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47786799 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410103615227904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12953326464000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410103633840128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874578421571584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784350854839717e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97115842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1864984472910730 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874578779950080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1007 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113457232 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423196921765376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12953326464000 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47790322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21459476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1861772694515712 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1861772677476352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1865131994636436 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97123474 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97123472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113456307 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210579779 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13094330254848 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1861772677476352 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 56413dŒ™G¿ 3888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12954339098112 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1852197404303360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410137062598656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874727033613824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97123472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113456307 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589772 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13094418824704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1865131637047074 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26331110 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0÷ƒ‹j'‰'ŒN H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874726675228160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210579779 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47790586 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423231374241280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚10514.151402822 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410137043986432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210579779 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13094330254848 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1861772677476352 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12954339098112 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1852197404303360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410137062598656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874727033613824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97123472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113456307 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589772 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13094418824704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1865131637047074 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26331110 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12953997751808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47790322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21459476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1861772694515712 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1861772677476352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1865131994636436 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97123474 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97123472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113456307 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13004021965824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868987487928320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113901707 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26432255 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868987504967680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424919627426816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97497442 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411775071766528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47977236 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13003680626688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1868987487928320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97497442 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113901707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1859376427225088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97497440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13004021965824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1872360265401988 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12983838176900 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1881991168555008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13003680626688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1881991526933504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424919734616064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1872360622991238 trainer/global_step‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881991168555008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47976960¿@FÑÅ ˜'‚P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113911684 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1881991526933504 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411775071766528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47977236 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13003680626688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1868987487928320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97497442 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚¸TÒ2C 113901707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1859376427225088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97497440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13004021965824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1872360265401988 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12983838176900 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1881991168555008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13003680626688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1881991526933504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424919734616064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1872360622991238 trainer/global_step‚12299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881991168555008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47976960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21544981 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26431982 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13004021965824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868987487928320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113901707 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26432255 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868987504967680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424919627426816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97497442 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1869134728957952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424954074734080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21546726 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1882139780597248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1869134745997312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47981007 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211415831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411808500525056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1882139780597248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97505072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13004693253632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211415831 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13145592821248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1859522937896960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12985189780624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47980732 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13004693253632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13005034599936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47980732 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13004693253632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13005034599936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1869134728957952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424954074734080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21546726 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1882139780597248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1869134745997312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47981007 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211415831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411808500525056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1882139780597248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97505072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13004693253632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211415831 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13145592821248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1859522937896960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12985189780624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1876349539409920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48168091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21632228 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1889403915538432 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356159 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1889403915538432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13054376128512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚10595.421685343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426642393683456 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13054717467648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413446491080704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1876349539409920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48168091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21632228 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1889403915538432 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356159 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1889403915538432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13054376128512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚10595.421685343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426642393683456 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13054717467648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413446491080704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48171940 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1876496780439552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1889552169195008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114365211 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212261876 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97886674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114375202 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48171688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13055730101760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114375202 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1879884294797452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13055730101760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13055388755456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413479919839232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13196843756544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13035465717530 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26537715 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426676854000640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21633973 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212251883 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1876496797478912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426676763595776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212261876 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚12349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114365211 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21633976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1876496780439552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13196915549184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1879883937208090 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1866848488529920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97886672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1889552527580672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1876496797478912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1889552527580672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843509373423636e+09 P+¥™­'ŠN KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426676763595776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_t†à­Uw$rain_batch_end/allocation.all.allocated‚ 212261876 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚12349 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114365211 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21633976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1876496780439552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13196915549184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1879883937208090 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1866848488529920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97886672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1889552527580672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1876496797478912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1889552527580672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843509373423636e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212251883 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48171940 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1876496780439552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1889552169195008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114365211 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212261876 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97886674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114375202 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48171688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13055730101760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114375202 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1879884294797452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13055730101760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13055388755456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413479919839232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13196843756544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13035465717530 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26537715 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426676854000640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21633973 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212251883 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1876496797478912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1896816662521856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415117929007104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48358501 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1896817020900352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13085411700606 _runtime‚10636.440238508 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1896817020900352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114810611 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1883711590891520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1887112923152254 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13105071630336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48358247 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896816662521856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13105412969472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98260642 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843509777465513e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21719478 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13247036998144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1874027511451648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114820588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415117947619328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591h^Fä¼'‚P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21719481 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1883711607930880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428365066903040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13085071150716 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213071251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1883711607930880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114820588 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98260642 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1896816662521856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415117929007104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48358501 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1896817020900352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13085411700606 _runtime‚10636.440238508 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1896817020900352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMoni©¾ï tor.on_train_batch_start/active.small_pool.freed‚ 114810611 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1883711590891520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1887112923152254 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13105071630336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48358247 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896816662521856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13105412969472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98260642 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843509777465513e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21719478 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13247036998144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1874027511451648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114820588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415117947619328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ûD9'Ù4½'/ _wandbruntime‚10637  _runtime‚10637ÔYíœ '¾'†N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13086422754440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428399523107840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13106084257280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48362264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1874174022123520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213097928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114829654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13248146729984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1883858831921152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428399418022912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21721226 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843509783633595e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087935 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1883858831921152 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1896965274564096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13106084257280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48361996 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1887260444877960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1883858848960512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13248060257280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13086082204438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1896964916178432 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98268272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13106425603584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98268274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21721223 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1896965274564096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98268272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114819663 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13086082204438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1896964916178432 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98268272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13106425603584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98268274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21721223 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1896965274564096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98268272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114819663 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13106084257280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48362264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1874174022123520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213097928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114829654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13248146729984 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1883858831921152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428399418022912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21721226 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843509783633595e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087935 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1883858831921152 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1896965274564096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13106084257280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48361996 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1887260444877960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1883858848960512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13248060257280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 Q&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23987415432929993 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6806157827377319 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024125011637806892 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.776597797870636 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22769968211650848 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980568885803223 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361343860626221 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7061432003974915 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9464071989059448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48742470145225525 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8766916394233704 +jet/accuracy_1_of_1‚0.33865636587142944 +jet/accuracy_2_of_2‚0.12247967720031738 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024752333760261536 trainer/global_step‚12399 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09705688059329987 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30493396520614624 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.50019326Äw .&978630066 *jet/accuracy_1_of_4‚0.9824040532112122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22210542857646942 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03267841041088104 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6797783970832825 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.618657648563385 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8593348264694214 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33170369267463684 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2101094126701355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21853110194206238 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9164679646492004 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7619745135307312 *jet/accuracy_1_of_2‚0.7732288837432861 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02110922522842884 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04451964423060417 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20998966693878174 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7928343415260315 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5818076133728027 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7293736934661865 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2801022231578827 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32740187644958496 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7122287154197693 particle/accuracy_1_of_3‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7validation_average_jet_accuracy‚0.37168964743614197 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7289826273918152 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05018262192606926 &particle/f_score‚0.608484148979187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955125451087952 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.618657648563385 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7914851903915405 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7509505748748779 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.715979814529419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 epoch‚30 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48562055826187134 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10452894121408463 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.676702618598938 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972004890441895 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6916708946228027 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22830383479595184 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2273012399673462 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2801022231578827 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8838963508605957 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2921008765697479 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6334195733070374 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 *jet/accuracy_2_of_4‚0.7618305683135986 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13722355663776398 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995419979095459 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7652280926704407 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7703346014022827 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *particle/specificity‚0.899133026599884 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704838752746582 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6603120565414429 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3644089102745056 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 % _timestamp‚1.7843509970296493e+09 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28690236806869507 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024519741535186768 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 -particle/accuracy_4_of_4‚0.95313560962677 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5639488101005554 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6060110926628113 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11255541443824768 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6428353190422058 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878772497177124 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2102186530828476 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6195608973503113 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4857271611690521 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08591856807470322 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7882353663444519 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023934906348586082 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5968092679977417 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714863538742065 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6322186589241028 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_2_of_3‚0.49870672821998596 (particle/accuracy‚0.8345850110054016 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35350099205970764 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6413846611976624 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6616795063018799 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15658295154571533 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6323683857917786 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1622130572795868 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02110922522842884 jet/accuracy_3_of_5‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0¬3¨œ=Ç'–{ @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980568885803223 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361343860626221 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7061432003974915 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9464071989059448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48742470145225525 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8766916394233704 +jet/accuracy_1_of_1‚0.33865636587142944 +jet/accuracy_2_of_2‚0.12247967720031738 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024752333760261536 trainer/global_step‚12399 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09705688059329987 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30493396520614624 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5001932978630066 *jet/accuracy_1_of_4‚0.9824040532112122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22210542857646942 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03267841041088104 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6797783970832825 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.618657648563385 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8593348264694214 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33170369267463684 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2101094126701355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21853110194206238 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9164679646492004 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7619745135307312 *jet/accuracy_1_of_2‚0.7732288837432861 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02110922522842884 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04451964423060417 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20998966693878174 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7928343415260315 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5818076133728027 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7293736934661865 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2801022231578827 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32740187644958496 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7122287154197693 particle/accuracy_1_of_3‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7validation_average_jet_accuracy‚0.37168964743614197 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7289826273918152 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05018262192606926 &particle/f_score‚0.608484148979187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955125451087952 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.618657648563385 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7914851903915405 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5337499976158142 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7509505748748779 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.715979814529419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 epoch‚30 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48562055826187134 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10452894121408463 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.676702618598938 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972004890441895 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6916708946228027 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22830383479595184 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2273012399673462 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2801022231578827 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8838963508605957 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2921008765697479 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6334195733070374 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 *jet/accuracy_2_of_4‚0.7618305683135986 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13722355663776398 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995419979095459 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7652280926704407 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7703346014022827 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *particle/specificity‚0.899133026599884 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704838752746582 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6603120565414429 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3644089102745056 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 % _timestamp‚1.7843509970296493e+09 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28690236806869507 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024519741535186768 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 -particle/accuracy_4_of_4‚0.95313560962677 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5639488101005554 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6060110926628113 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11255541443824768 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6428353190422058 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878772497177124 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2102186530828476 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6195608973503113 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4857271611690521 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08591856807470322 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7882353663444519 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023934906348586082 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5968092679977417 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714863538742065 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6322186589241028 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_2_of_3‚0.49870672821998596 (particle/accuracy‚0.8345850110054016 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35350099205970764 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6413846611976624 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6616795063018799 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15658295154571533 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6323683857917786 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1622130572795868 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02110922522842884 jet/accuracy_3_of_5‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23468485474586487 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8766916394233704 ,jet/accuracy_3_of_3‚0.040348175913095474 )loss/val_total_loss‚7.703242327490534 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9944398999214172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7914471626281738 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961845874786377 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27172937989234924 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 _runtime‚10655.723778225 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7445862889289856 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7182957530021667 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 _step‚1022 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21792104840278625 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23042242228984833 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03316659852862358 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.713996410369873 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.5985674858093262 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6998307108879089 +jet/accuracy_3_of_4‚0.04682844132184982 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8495086431503296 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8593348264694214 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04778636246919632 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11966387182474136 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8457860350608826 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2806329131126404 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23934419453144073 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7826341986656189 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2908868193626404 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2807469069957733 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.994434654712677 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6594341993331909 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12185841798782349 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3072344660758972 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871999621391296 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23987415432929993 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6806157827377319 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024125011637806892 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.776597797870636 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22769968211650848þzm>‘È'‚(:† ž’ìÒ˦–” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÏÈ¥òÉ'‚(:ç ž’ìÒ†¦¥•) disk./.usagePercent‚39.49244929330452 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.17555046081543# proc.memory.rssMB‚ 2738.36328125 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 17773.3515625* proc.memory.percent‚16.713643074035645 cpu‚26.1802625627795 network.recv‚ 2.865241e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚29004.6806640625 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚588.4660339355469 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 320.00390625# disk.nvme0c0n1.in‚ 17773.3515625 disk.sda.out‚ 320.00390625 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 462.7578125 disk.dm-1.in‚ 0.1328125' disk.nvme0c0n1.out‚29004.6806640625 disk.sda3.out‚ 320.00390625 network.sent‚ 1.98217e+07 disk.rbd1.out‚ 37.0791015625 disk.dm-0.in‚ 17773.3515625" disk.dm-0.out‚29004.6806640625ŸÆªwžÊ'‚(:“ ­’ìÒ¹Ôú” gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚272.752' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.550399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yªqúË'‚(:ï ­’ìÒ£–µ– disk.dm-1.in‚ 0.1328125 disk.sda.out‚ 320.16015625* proc.memory.percent‚16.713643074035645 network.sent‚ 1.9879233e+07 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2738.36328125( disk.nvme0c0n1.out‚29008.92431640625( proc.memory.availableMB‚ 462.83203125) disk./.usagePercent‚39.49244417315383 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚97.17509746551514 cpu‚26.180051048290736 network.recv‚ 2.869634e+06! disk.nvme0n1.in‚ 17773.3671875 disk.rbd1.out‚ 37.0791015625 disk.dm-1.out‚ 320.16015625# disk.nvme0c0n1.in‚ 17773.3671875# disk.dm-0.out‚29008.92431640625 disk.sda.in‚ 0.1328125$ disk./.usageGB‚588.4659576416016 disk.sda3.out‚ 320.16015625 disk.drbd2165.in‚0 disk.dm-0.in‚ 17773.3671875 disk.sda2.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚29008.92431640625 disk.sda3.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0Z.…2“Ì'‚(:ˆ ¼’ìÒÐÏå“ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ËŸ“ôÍ'‚(:é ¼’ìÒ•í•– disk.sda.out‚ 320.3828125! disk.nvme0n1.in‚ 17773.3671875 disk.sda1.out‚0 network.sent‚ 1.9884217e+07 disk.sda2.out‚0% disk.nvme0n1.out‚29012.2958984375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 462.45703125 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.17738628387451 network.recv‚ 2.873069e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 17773.3671875 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2738.36328125$ disk./.usageGB‚588.4643707275391 disk.sda3.out‚ 320.3828125 disk.rbd1.out‚ 37.0791015625 disk.dm-0.in‚ 17773.3671875* proc.memory.percent‚16.713643074035645 cpu‚26.179683841762998) disk./.usagePercent‚39.49233767401958 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 320.3828125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚29012.2958984375" disk.dm-0.out‚29012.2958984375 disk.sda2.in‚0 disk.rbd0.out‚0Sšø°4Î'/ _wandbruntime‚10696  _runtime‚10696YÀ FÌÏ'Æ ' lr-AdamW/pg1‚0.00014533812993029624 ' lr-AdamW/pg2‚0.00014533812993029624 trainer/global_step‚12449 % _timestamp‚1.7843510377505996e+09 _runtime‚10696.446553892 _step‚10232ÚPôÑÐ'Ë _runtime‚10696.446553892 _step‚1023 ' lr-AdamW/pg1‚0.00014533812993029624 ' lr-AdamW/pg2‚0.00014533812993029624 trainer/global_step‚12449 % _timestamp‚1.7843510377505996e+09ÿOÈeY4Ñ'/ _wandbruntime‚10696  _runtime‚10696»§fJ4 Ò'ûO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1883261708259328 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48583652 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430121780184576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762906 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115335385 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1906177067739648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_deY7/²Ívice_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1906177067739648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 trainer/global_step‚12449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13173500337664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416805297645568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416805279033344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1883261691219968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13316577145344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1893003209023488 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26774671 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214098291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13316501151232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906176709361152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13173500337664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1896415098582821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115345362 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0œ{²ø(Ó'€P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1906176709361152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48583922 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115345362 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843510377538233e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1893003209023488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13173841676800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1893003226062848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98762908 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13153747912743 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98762908 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214108270 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26774671 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214098291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13316501151232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906176709361152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13173500337664 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1896415098582821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115345362 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1883261708259328 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48583652 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430121780184576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762906 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115335385 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1906177067739648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1906177067739648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 trainer/global_step‚12449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13173500337664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416805297645568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416805279033344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1883261691219968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13316577145344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1893003209023488 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114975 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115354428 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174854310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13174854310912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13174512964608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1896562620308415 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚10697.061210585 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _step‚1025 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21810993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115344437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13154418416575 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115354428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1906324963017728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98770540 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1906325321403392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48587477 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1883408201891840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1893150450053120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430156333494272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430156232603648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214124968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1893150467092480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664€ô¦´ÉÖ'ŒN OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48587731 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26776484 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13317524811776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13154758966577 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚12449 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1906325321403392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48587477 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1883408201891840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1893150450053120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430156333494272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430156232603648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214124968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1893150467092480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114975 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115354428 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174854310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13174854310912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.fr¥¬¡Ieed‚13174512964608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1896562620308415 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚10697.061210585 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _step‚1025 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21810993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115344437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13154418416575 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115354428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1906324963017728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98770540 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99144506 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1890587241852928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚1028 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115789837 _runtime‚10737.298370607 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1900365277544448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1913589456344576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13224537178624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115799814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115789837 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888ó9/¿(ã'P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431844466375680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13367840090112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13224195839488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21896498 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99144506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418476735571968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚12499 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99144508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784351078604528e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214944322 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1900365260505088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1913589814723072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26878703 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13224537178624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13367749415936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1890587224813568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1903791248663329 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1913589814723072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99144508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13224195839488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13204364399651 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1913589456344576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48774947 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1913589456344576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13224537178624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115799814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115789837 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115799814 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99144506 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1890587241852928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚1028 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115789837 _runtime‚10737.298370607 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1900365277544448 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1900512501534720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3583856ƒa 64 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13368766627328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13225549812736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961020 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1913738068386816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1900512501534720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99152140 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13205375453485 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1890733735485440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99152138 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48778991 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115808880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1903939127978285 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48778718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1903938770388923 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1890733752524800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21898243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26880475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚10737.916388587 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13225549812736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1913738068386816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115808880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115798889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961020 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214951027 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1900512518574080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431879027916288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843510792225826e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13225208466432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99152138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431878912345600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21898246 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99152140 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26880745 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418510145718272 _step‚1029 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169ØŸÚ 'æ'ŒN J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48778991 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115808880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1903939127978285 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48778718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1903938770388923 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1890733752524800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21898243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26880475 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚10737.916388587 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13225549812736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1913738068386816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115808880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115798889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961020 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214951027 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1900512518574080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431879027916288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843510792225826e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13225208466432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99152138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431878912345600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21898246 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99152140 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26880745 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418510145718272 _step‚1029 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1900512501534720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13368766627328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13225549812736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961020 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1913738068386816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1900512501534720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99152140 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13205375453485 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1890733735485440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99152138 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0… Ýü©4ç'/ _wandbruntime‚10737  _runtime‚10737ÿ[°¯ è'‡ 2loss/SRqqt1/assignment_loss‚0.5242351116004966 3loss/SRqqt2/assignment_loss‚0.46610588189982377 .loss/FRt2/detection_loss‚1.513109625652864 1loss/SRqqt1/detection_loss‚0.8275713043843919 /loss/FBt1/detection_loss‚0.4315633098206793 _runtime‚10737.919355609 %loss/total_loss‚7.502269733006292 0loss/FBt1/assignment_loss‚0.0029438044907582 1loss/SRqqt2/detection_loss‚0.5882987292532375 0loss/FRt1/assignment_loss‚1.5317053578951914 3loss/FBt2/assignment_loss‚0.0012099542536049919 /loss/FRt1/detection_loss‚0.5692247842926954 0loss/FBt2/detection_loss‚0.40445905775590235 epoch‚31 trainer/global_step‚12499 % _timestamp‚1.7843510792260337e+09 _step‚1030 0loss/FRt2/assignment_loss‚0.6418428117066474øKJé'Œ %loss/total_loss‚7.502269733006292 0loss/FBt1/assignment_loss‚0.0029438044907582 1loss/SRqqt2/detection_loss‚0.5882987292532375 0loss/FRt1/assignment_loss‚1.5317053578951914 3loss/FBt2/assignment_loss‚0.0012099542536049919 /loss/FRt1/detection_loss‚0.5692247842926954 0loss/FBt2/detection_loss‚0.40445905775590235 epoch‚31 trainer/global_step‚12499 % _timestamp‚1.7843510792260337e+09 _step‚1030 0loss/FRt2/assignment_loss‚0.6418428117066474 2loss/SRqqt1/assignment_loss‚0.5242351116004966 3loss/SRqqt2/assignment_loss‚0.46610588189982377 .loss/FRt2/detection_loss‚1.513109625652864 1loss/SRqqt1/detection_loss‚0.8275713043843919 /loss/FBt1/detection_loss‚0.4315633098206793 _runtime‚10737.919355609†9àÅp‘ê'‚(:† é’ì҈㕔 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.094' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0188 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ©ôë'‚(:é é’ìÒó±­—$ memory_percent‚97.17674255371094 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17839.62890625# disk.dm-0.out‚29029.74951171875 disk.sda.out‚ 321.2734375 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 37.2041015625 disk.rbd0.out‚0 disk.dm-0.in‚17839.62890625$ proc.memory.availableMB‚462.5625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 network.sent‚ 1.9970227e+07 disk.sda1.in‚0 disk.dm-1.out‚ 321.2734375 disk.sda.in‚ 0.1328125! proc.memory.rssMB‚ 2738.421875 network.recv‚ 2.884676e+06 disk.drbd2165.out‚0& disk.nvme0n1.out‚29029.74951171875 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0" disk.nvme0n1.in‚17839.62890625 disk.sda3.out‚ 321.2734375$ disk./.usageGB‚588.4653663635254 disk.rbd0.in‚0) disk./.usagePercent‚39.49240449198602( disk.nvme0c0n1.out‚29029.74951171875 disk.sda2.out‚0* proc.memory.percent‚16.714000701904297 cpu‚26.179205567143597foNÍ‘ì'‚(:† ø’ìÒŽØÜ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚286.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒØ‹töí'‚(:ë ø’ìÒ͌ї network.recv‚ 2.889327e+06 disk.drbd2165.in‚0 disk.sda.out‚ 321.69921875! proc.memory.rssMB‚ 2738.421875* proc.memory.percent‚16.714000701904297 disk.sda2.in‚0 network.sent‚ 2.002033e+07 disk.sda1.out‚0' disk.nvme0c0n1.out‚29033.6474609375" disk.dm-0.out‚29033.6474609375 proc.cpu.threads‚17 disk.dm-1.out‚ 321.69921875 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125$ disk.nvme0c0n1.in‚17839.62890625( proc.memory.availableMB‚ 462.02734375 disk.rbd1.out‚ 37.2041015625 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0% disk.nvme0n1.out‚29033.6474609375 disk.sda3.out‚ 321.69921875 cpu‚26.17900130242493 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚17839.62890625$ disk./.usageGB‚588.4655990600586) disk./.usagePercent‚39.49242010844561 disk.rbd1.in‚ 2.01953125 disk.drbd2165.out‚0" disk.nvme0n1.in‚17839.62890625$ memory_percent‚97.18000888824463¾¨5—“î'‚(:ˆ ‡“ìÒ–Æñ“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚274.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ªI&úï'‚(:ï ‡“ìÒÄس– disk.sda.out‚ 322.44921875 disk.sda2.out‚0 cpu‚26.178874627643964 disk.rbd1.in‚ 2.01953125" disk.nvme0n1.in‚17839.62890625& proc.memory.availableMB‚ 462.515625! proc.memory.rssMB‚ 2738.421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29050.26904296875# disk.dm-0.out‚29050.26904296875 network.recv‚ 2.892975e+06* disk./.usagePercent‚39.283861778624015 disk.rbd1.out‚ 37.2666015625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125* proc.memory.percent‚16.714000701904297 proc.cpu.threads‚17 disk.sda3.out‚ 322.44921875 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 2.0025416e+07$ disk./.usageGB‚585.3579292297363 disk.dm-0.in‚17839.62890625& disk.nvme0n1.out‚29050.26904296875$ memory_percent‚97.17702865600586 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 322.44921875$ disk.nvme0c0n1.in‚17839.62890625pÈDC4ð'/ _wandbruntime‚10778  _runtime‚10778EÅÚŽËñ'Å _step‚1031 ' lr-AdamW/pg1‚0.00014198074515783636 ' lr-AdamW/pg2‚0.00014198074515783636 trainer/global_step‚12549 $ _timestamp‚1.784351119767807e+09 _runtime‚10778.461804869D™MÐò'Ê ' lr-AdamW/pg1‚0.00014198074515783636 ' lr-AdamW/pg2‚0.00014198074515783636 trainer/global_step‚12549 $ _timestamp‚1.784351119767807e+09 _runtime‚10778.461804869 _step‚1031‡„ÚvÆ4ó'/ _wandbruntime‚10778  _runtime‚10778+ìù.ô'úO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420148173498368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1921002203328000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1897912775446528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13418983654912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116254266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚12549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13274891341312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784351119769071e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13254640336669 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_Åö`ÕÒbatch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48966379 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215770395 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116254266 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1907727311986688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1907727329026048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1907727311986688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚420148154886144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1911167398743837 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13254980886559 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1911167756333087 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1897912758407168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26982628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26982364 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ã0œª(õ'ÿO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48966379 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215770395 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116254266 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1907727311986688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1907727329026048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1907727311986688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚420148154886144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1911167398743837 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13254980886559 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1911167756333087 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1897912758407168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26982628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26982364 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420148173498368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1921002203328000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1897912775446528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13418983654912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116254266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚12549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13274891341312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784351119769071e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13254640336669ˆÑXŒz4ö'/ _wandbruntime‚10779  _runtime‚10779Øáý '÷'‡N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13276245314560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116253341 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1907874553016320 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1898059286118400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚10779.083288198 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1907874553016320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116253341 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1921150815370240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1911314920469431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48969865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1911315278058793 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99533740 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1907874570055680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433601586419712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433601701990400 _step‚1033 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚420181583644672 % _timestamp‚1.7843511203893433e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21985496 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1921150815370240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13275903968256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116263332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21985493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13275903968256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215797072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99533740 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13420099733504 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48969865 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1911315278058793 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99533740 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1907874570055680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433601586419712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433601701990400 _step‚1033 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚420181583644672 % _timestamp‚1.7843511203893433e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21985496 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1921150815370240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13275903968256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116263332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21985493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13275903968256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215797072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99533740 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13420099733504 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1898059286118400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚10779.083288198 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1907874553016320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116253341 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1921150815370240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1911314920469431 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13325928182272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49157204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27085936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216616426 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421819592812544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99907706 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27086203 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚12599 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928414950311424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99907706 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13470294914048 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1905238309040128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928415308689920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305256823577 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1915089363468288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435289790760960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚10819.587567037 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116698741 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928414950311424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7843511608916721e+09 _step‚1036 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13325928182272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49156934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13325586843136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99907708áw×(‡(‚P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99907708 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1915089380507648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22071001 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116698741 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22070998 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13325586843136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13305597373467 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1905238292000768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99907708 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421819611424768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13470197948416 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1918543548824345 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13325928182272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49157204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27085936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216616426 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421819592812544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99907706 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27086203 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚12599 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928414950311424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99907706 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13470294914048 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1905238309040128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928415308689920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305256823577 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1915089363468288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435289790760960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚10819.587567037 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116698741 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928414950311424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7843511608916721e+09 _step‚1036 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13325928182272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49156934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13325586843136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101ŒcËš}4ˆ(/ _wandbruntime‚10820  _runtime‚10820y<°É‰(‡N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623131 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1915236604497920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1928563203968000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13471219494912 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49160651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435324354539520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13326599470080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99915338 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27088163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1905384819712000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27087908 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633124 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1918691428139301 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13306267877299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928563562353664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116707793 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1905384802672640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843511615101073e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1928563562353664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116707793 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚1037 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1915236621537280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421853040183296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13306608427301 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633124 _runtime‚10820.203842295 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99915338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13471314356224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1915236604497920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216623131 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99915340 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116717784 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19186910705499391*Ój'Š(ŒN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116717784 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1918691070549939 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623131 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1915236604497920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1928563203968000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13471219494912 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49160651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435324354539520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13326599470080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99915338 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27088163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1905384819712000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27087908 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633124 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1918691428139301 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13306267877299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928563562353664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116707793 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1905384802672640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843511615101073e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1928563562353664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116707793 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚1037 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1915236621537280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421853040183296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13306608427301 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633124 _runtime‚10820.203842295 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99915338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13471314356224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1915236604497920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216623131 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99915340 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616—Ù¨ø4‹(/ _wandbruntime‚10820  _runtime‚108203N—Ô Œ(„ 0loss/FRt1/assignment_loss‚1.5662193331192797 0loss/FRt2/assignment_loss‚0.6293204225380135 /loss/FRt1/detection_loss‚0.5329646522282627 1loss/SRqqt2/detection_loss‚0.6264646512853862 /loss/FBt1/detection_loss‚0.5383720125096876 /loss/FBt2/detection_loss‚0.4027439746903256 trainer/global_step‚12599 3loss/SRqqt2/assignment_loss‚0.44136921974855825 1loss/FBt1/assignment_loss‚0.08352040145641318 %loss/total_loss‚7.631856927779435 # _timestamp‚1.78435116151383e+09 3loss/SRqqt1/assignment_loss‚0.48683137335326576 /loss/FRt2/detection_loss‚1.3639009744692143 _step‚1038 1loss/FBt2/assignment_loss‚0.01014057004416594 1loss/SRqqt1/detection_loss‚0.9500093423368625 _runtime‚10820.20710786 epoch‚31¢ÿé”(‰ 1loss/FBt2/assignment_loss‚0.01014057004416594 1loss/SRqqt1/detection_loss‚0.9500093423368625 _runtime‚10820.20710786 epoch‚31 0loss/FRt1/assignment_loss‚1.5662193331192797 0loss/FRt2/assignment_loss‚0.6293204225380135 /loss/FRt1/detection_loss‚0.5329646522282627 1loss/SRqqt2/detection_loss‚0.6264646512853862 /loss/FBt1/detection_loss‚0.5383720125096876 /loss/FBt2/detection_loss‚0.4027439746903256 trainer/global_step‚12599 3loss/SRqqt2/assignment_loss‚0.44136921974855825 1loss/FBt1/assignment_loss‚0.08352040145641318 %loss/total_loss‚7.631856927779435 # _timestamp‚1.78435116151383e+09 3loss/SRqqt1/assignment_loss‚0.48683137335326576 /loss/FRt2/detection_loss‚1.3639009744692143 _step‚1038ŽXí+U“Ž(‚(:ˆ ÓìÒÕéó“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N8v4ò(‚(:ç ÓìÒåç — disk.sda1.in‚0$ memory_percent‚97.17981815338135 network.sent‚ 2.0106335e+07 disk.sda2.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 17905.34375 disk.dm-0.in‚ 17905.34375 disk.nvme0n1.in‚ 17905.34375 disk.rbd1.in‚ 2.01953125 cpu‚26.177859014434095$ disk./.usageGB‚585.3572463989258 disk.dm-1.out‚ 323.81640625 disk.drbd2165.out‚0* proc.memory.percent‚16.714000701904297 disk.sda.out‚ 323.81640625 disk.sda2.in‚0 disk.rbd1.out‚ 37.4541015625 disk.rbd0.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2738.421875 network.recv‚ 2.913931e+06( disk.nvme0c0n1.out‚29093.35205078125# disk.dm-0.out‚29093.35205078125 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚29093.35205078125 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0( proc.memory.availableMB‚ 462.05859375 disk.sda3.out‚ 323.81640625 proc.cpu.threads‚17) disk./.usagePercent‚39.28381595327539OPÐ}“(‚(:ˆ Ò“ìÒ°”ó“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cßÜ'î‘(‚(:ã Ò“ìÒë᎗ disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 17978.171875! disk.dm-0.out‚29097.845703125 cpu‚26.177660306278696 disk.dm-1.out‚ 324.02734375 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125( proc.memory.availableMB‚ 462.04296875) disk./.usagePercent‚39.28381390521511 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125! proc.memory.rssMB‚ 2738.421875 disk.rbd0.in‚0 disk.sda.out‚ 324.02734375 disk.sda2.out‚0$ disk.nvme0n1.out‚29097.845703125$ memory_percent‚97.17991352081299 network.sent‚ 2.0169368e+07 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚29097.845703125 disk.rbd1.out‚ 37.4541015625 proc.cpu.threads‚17$ disk./.usageGB‚585.3572158813477 disk.rbd0.out‚0* proc.memory.percent‚16.714000701904297 disk.sda3.out‚ 324.02734375 network.recv‚ 2.91831e+06 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.dm-0.in‚ 17978.171875 disk.nvme0n1.in‚ 17978.171875}¿×’(‚(:’ á“ìÒ¨µ·“ gpu.0.gpu‚35.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.575' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.114999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ײWí“(‚(:â á“ìÒÖ Û• disk.sda2.out‚0 cpu‚26.177353676933222 disk.dm-0.in‚ 17978.1875& disk.nvme0n1.out‚29103.86474609375 disk.sda3.out‚ 324.26953125 disk.rbd1.out‚ 37.5166015625 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0$ memory_percent‚97.17988967895508 network.sent‚ 2.0174347e+07 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚29103.86474609375& proc.memory.availableMB‚ 462.046875! proc.memory.rssMB‚ 2738.421875$ disk./.usageGB‚585.3572540283203# disk.dm-0.out‚29103.86474609375 disk.sda2.in‚0 disk.dm-1.out‚ 324.26953125 disk.drbd2165.out‚0* proc.memory.percent‚16.714000701904297) disk./.usagePercent‚39.28381646529046 disk.sda.in‚ 0.1328125 disk.sda.out‚ 324.26953125 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 17978.1875 proc.cpu.threads‚17 network.recv‚ 2.921792e+06 disk.nvme0n1.in‚ 17978.1875 disk.dm-1.in‚ 0.1328125Ýø)4”(/ _wandbruntime‚10860  _runtime‚10860äÂP Ì•(Æ % _timestamp‚1.7843512018318086e+09 _runtime‚10860.525660967 _step‚1039 ' lr-AdamW/pg1‚0.00013864457015872614 ' lr-AdamW/pg2‚0.00013864457015872614 trainer/global_step‚12649Ý)KQÑ–(Ë trainer/global_step‚12649 % _timestamp‚1.7843512018318086e+09 _runtime‚10860.525660967 _step‚1039 ' lr-AdamW/pg1‚0.00013864457015872614 ' lr-AdamW/pg2‚0.00013864457015872614{Ïʬ4—(/ _wandbruntime‚10860  _runtime‚10860böU(˜(P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1935827697294848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13376282344960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437012333127680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13521302388736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1925919698904853 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1922451414949888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1922451414949888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217452478 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423491030738944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13521386771456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1922451431989248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100289306 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100289306 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217442499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13376623684096 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1925920056494103 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1912563842633728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49347554 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117163170 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1922451431989248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚10860.529388124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935827697294848 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437012436122624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117153193 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49347814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423491049351168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22158248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935828055673344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100289308 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935828055673344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7843512018355217e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13356213860375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13376282344960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27189563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1912563825594368 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117153193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚12649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22158251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13355873310485 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13376623684096 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Æ™+D”™(†P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100289308 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1935827697294848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13376282344960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437012333127680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13521302388736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1925919698904853 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1922451414949888 ]HDeviceStatsMonitor.on_train_batch_start/allocf~1Ëx ated_bytes.large_pool.freed‚1922451414949888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217452478 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423491030738944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13521386771456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1922451431989248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100289306 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100289306 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217442499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13376623684096 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1925920056494103 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1912563842633728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49347554 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117163170 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1922451431989248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚10860.529388124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935827697294848 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437012436122624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117153193 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49347814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423491049351168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22158248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935828055673344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100289308 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935828055673344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7843512018355217e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13356213860375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13376282344960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27189563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1912563825594368 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117153193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚12649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22158251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13355873310485 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13376623684096 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117172236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27191581 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1922598655979520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1922598655979520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49351321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423524459497472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚1041 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217469176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22159993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437046892114432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100296938 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13377636318208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117162245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13522414004736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1912710336266240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13522331726336 trainer/global_step‚12649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117172236 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117162245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1922598673018880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22159996 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459183 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843512024529395e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217459183 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13377294971904 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1935975950951424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49351577 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423524478109696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13357224914209 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1926067220630447 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100296940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13377636318208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10861.146739018 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1912710353305600Qc{•'œ(N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13377294971904 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1922598673018880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117172236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27191581 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1922598655979520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1922598655979520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49351321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423524459497472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚1041 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217469176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22159993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437046892114432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100296938 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13377636318208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117162245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13522414004736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1912710336266240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13522331726336 trainer/global_step‚12649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117172236 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117162245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1922598673018880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22159996 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459183 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843512024529395e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217459183 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13377294971904 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1935975950951424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49351577 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423524478109696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13357224914209 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1926067220630447 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100296940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13377636318208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10861.146739018 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1912710353305600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976‘ÉYì4(/ _wandbruntime‚10861  _runtime‚10861'‡† ž(‡ /loss/FRt1/assignment_loss‚1.451419286851065 2loss/SRqqt2/assignment_loss‚0.4380265994922635 /loss/FRt2/detection_loss‚1.3153886948105895 1loss/SRqqt1/detection_loss‚0.7886828906736394 0loss/FBt2/detection_loss‚0.43102726183237144 %loss/total_loss‚7.182518253758432 _runtime‚10861.148930866 1loss/FBt1/assignment_loss‚0.04930911723537303 2loss/FBt2/assignment_loss‚0.002531235043247471 epoch‚31 1loss/SRqqt2/detection_loss‚0.5733664492469818 0loss/FBt1/detection_loss‚0.45688266280156214 trainer/global_step‚12649 % _timestamp‚1.7843512024556088e+09 _step‚1042 0loss/FRt2/assignment_loss‚0.6070321087087138 2loss/SRqqt1/assignment_loss‚0.5225558817411734 /loss/FRt1/detection_loss‚0.5462960653214509²¶Ç2Ÿ(Œ /loss/FRt1/detection_loss‚0.5462960653214509 /loss/FRt1/assignment_loss‚1.451419286851065 2loss/SRqqt2/assignment_loss‚0.4380265994922635 /loss/FRt2/detection_loss‚1.3153886948105895 1loss/SRqqt1/detection_loss‚0.7886828906736394 0loss/FBt2/detection_loss‚0.43102726183237144 %loss/total_loss‚7.182518253758432 _runtime‚10861.148930866 1loss/FBt1/assignment_loss‚0.04930911723537303 2loss/FBt2/assignment_loss‚0.002531235043247471 epoch‚31 1loss/SRqqt2/detection_loss‚0.5733664492469818 0loss/FBt1/detection_loss‚0.45688266280156214 trainer/global_step‚12649 % _timestamp‚1.7843512024556088e+09 _step‚1042 0loss/FRt2/assignment_loss‚0.6070321087087138 2loss/SRqqt1/assignment_loss‚0.5225558817411734’£l/Ù“ (‚(:ˆ ð“ìÒ­œ­“ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸I ˆå¡(‚(:Ú ð“ìÒ»ýÑ—( proc.memory.availableMB‚ 461.79296875 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚26.177008395644517$ memory_percent‚97.18143939971924 disk.rbd1.out‚ 37.5166015625 network.sent‚ 2.0179321e+07 disk.sda3.out‚ 324.71484375 disk.dm-0.in‚ 17978.1875# disk./.usageGB‚585.359489440918 disk.nvme0c0n1.in‚ 17978.1875 disk.sda.in‚ 0.1328125 disk.sda.out‚ 324.71484375 disk.nvme0n1.in‚ 17978.1875 disk.rbd0.out‚0 disk.dm-1.out‚ 324.71484375 disk.drbd2165.in‚0* proc.memory.percent‚16.714000701904297 network.recv‚ 2.924945e+06 disk.sda2.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚29107.12890625) disk./.usagePercent‚39.28396648570553 disk.sda2.out‚0 disk.dm-1.in‚ 0.1328125! proc.memory.rssMB‚ 2738.421875 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125# disk.nvme0n1.out‚29107.12890625 disk.rbd0.in‚0 disk.dm-0.out‚29107.12890625G©¡Y“¢(‚(:ˆ ÿ“ìÒã´Æ“ gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×bð÷‚£(‚(:Ô ÿ“ìÒîµ— network.recv‚ 2.93008e+06) disk./.usagePercent‚39.28394933320073 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 37.5791015625 disk.sda1.in‚0$ disk./.usageGB‚585.3592338562012% disk.nvme0n1.out‚29111.9345703125 disk.rbd0.in‚0 network.sent‚ 2.0243015e+07 disk.nvme0n1.in‚ 17978.1875 disk.sda3.out‚ 325.15625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.714000701904297$ memory_percent‚97.18141555786133 disk.nvme0c0n1.in‚ 17978.1875 disk.sda3.in‚ 0.1328125! proc.memory.rssMB‚ 2738.421875 cpu‚26.17672048099531& proc.memory.availableMB‚ 461.796875 disk.drbd2165.in‚0 disk.dm-0.in‚ 17978.1875" disk.dm-0.out‚29111.9345703125 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0' däÔ÷]isk.nvme0c0n1.out‚29111.9345703125 disk.sda.out‚ 325.15625 disk.dm-1.out‚ 325.15625o#A4¤(/ _wandbruntime‚10901  _runtime‚10901Ç.ÏÈ¥( _runtime‚10901.587548488 _step‚1043 % lr-AdamW/pg1‚0.000136984704384115 % lr-AdamW/pg2‚0.000136984704384115 trainer/global_step‚12699 % _timestamp‚1.7843512428907676e+09Àۃͦ(Ç % lr-AdamW/pg1‚0.000136984704384115 % lr-AdamW/pg2‚0.000136984704384115 trainer/global_step‚12699 % _timestamp‚1.7843512428907676e+09 _runtime‚10901.587548488 _step‚1043“›Ä4§(/ _wandbruntime‚10901  _runtime‚109017ê¶(¨(€P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1929813466431488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27293387 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1929813466431488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13572461127680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1943240444278272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13427319185920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100670906 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425162468665344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13406830347283 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843512428947868e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218278551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22245501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438734948405248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1919889359187968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1943240802656768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1933296206574611 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚10901.58860359 _step‚1044 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1929813483470848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13427319185920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218288530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438734839118848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100670908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13426977846784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100670906 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1943240802656768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13426977846784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1933295848985361 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943240444278272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425162487277568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117617622 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1933295848985361 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943240444278272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425162487277568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117617622 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438734948405248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1919889359187968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1943240802656768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1933296206574611 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚10901.58860359 _step‚1044 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1929813483470848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13427319185920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218288530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438734839118848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100670908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13426977846784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100670906 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1943240802656768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13426977846784 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305228 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13407841401117 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117616697 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100678538 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100678538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22247246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚10902.205064531 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27295430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218305228 trainer/global_step‚12699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚438769280921088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1920035886899200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1929960707461120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117626688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425195916036096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚1045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49542673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27295699 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1929960707461120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13573478358528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13427990473728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49542945 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1933443728300317 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1933443370710955 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117626688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13573383497216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100678540 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1929960724500480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13427990473728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22247243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943389056320512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218295235 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117616697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438769394394624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943388697934848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13407500851115 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13428331820032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13428331820032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218295235 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425195897423872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100678540 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚1045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49542673 VIDeviceStatsMonitog]àŒ !r.on_train_batch_end/inactive_split.small_pool.allocated‚27295699 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1929960707461120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13573478358528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13427990473728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49542945 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1933443728300317 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1933443370710955 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117626688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13573383497216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100678540 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1929960724500480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13427990473728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22247243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943389056320512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218295235 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117616697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438769394394624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943388697934848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13407500851115 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13428331820032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13428331820032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218295235 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425195897423872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100678540 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305228 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13407841401117 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117616697 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100678538 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100678538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22247246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚10902.205064531 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27295430•x²HŽ4­(/ _wandbruntime‚10902  _runtime‚10902D} ®(† /loss/FRt2/assignment_loss‚0.536886222485046 /loss/FBt2/detection_loss‚0.5961961022190555 epoch‚31 _runtime‚10902.208211622 2loss/SRqqt1/assignment_loss‚0.5032662293377981 3loss/FBt1/assignment_loss‚4.259723139529318e-05 1loss/SRqqt2/detection_loss‚0.5779625667487057 %loss/total_loss‚7.280205288172434 % _timestamp‚1.7843512435149174e+09 _step‚1046 2loss/SRqqt2/assignment_loss‚0.4415769047225127 .loss/FRt2/detection_loss‚1.304116255568831 1loss/SRqqt1/detection_loss‚0.8602408384224252 .loss/FBt1/detection_loss‚0.472371344421325 trainer/global_step‚12699 /loss/FRt1/assignment_loss‚1.444824179257352 4loss/FBt2/assignment_loss‚0.00016424690452162915 /loss/FRt1/detection_loss‚0.5425578008534659\¢Nì¯(‹ .loss/FRt2/detection_loss‚1.304116255568831 1loss/SRqqt1/detection_loss‚0.8602408384224252 .loss/FBt1/detection_loss‚0.472371344421325 trainer/global_step‚12699 /loss/FRt1/assignment_loss‚1.444824179257352 4loss/FBt2/assignment_loss‚0.00016424690452162915 /loss/FRt1/detection_loss‚0.5425578008534659 /loss/FRt2/assignment_loss‚0.536886222485046 /loss/FBt2/detection_loss‚0.5961961022190555 epoch‚31 _runtime‚10902.208211622 2loss/SRqqt1/assignment_loss‚0.5032662293377981 3loss/FBt1/assignment_loss‚4.259723139529318e-05 1loss/SRqqt2/detection_loss‚0.5779625667487057 %loss/total_loss‚7.280205288172434 % _timestamp‚1.7843512435149174e+09 _step‚1046 2loss/SRqqt2/assignment_loss‚0.4415769047225127–üÍ…ž°(‚(:“ Ž”ìÒÉíö“ gpu.0.gpu‚44.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦}@þï±(‚(:ä Ž”ìÒ½Üé— disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.nvme0n1.in‚ 18047.3125 disk.sda3.out‚ 325.57421875 proc.cpu.threads‚17* disk./.usagePercent‚39.283959061487025# disk.dm-0.out‚29115.40283203125 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 2.0251384e+07 network.recv‚ 2.940211e+06$ disk./.usageGB‚585.3593788146973 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29115.40283203125 disk.sda.in‚ 0.1328125 disk.sda1.in‚0 disk.dm-1.out‚ 325.57421875 disk.nvme0c0n1.in‚ 18047.3125 disk.dm-0.in‚ 18047.3125 disk.sda2.out‚0& disk.nvme0n1.out‚29115.40283203125 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0$ memory_percent‚97.18234539031982 cpu‚26.17652178389029 disk.sda.out‚ 325.57421875! proc.memory.rssMB‚ 2738.421875* proc.memory.percent‚16.714000701904297( proc.memory.availableMB‚ 461.64453125 disk.rbd1.out‚ 37.5791015625[Z}i“²(‚(:ˆ ”ìÒá»å“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚229.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.8978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á\)Aì³(‚(:á ”ìÒ‡øš• disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.dm-0.in‚ 18047.3125" disk.dm-0.out‚29121.3505859375 disk.sda.out‚ 325.91796875 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 325.91796875 disk.dm-1.out‚ 325.91796875* proc.memory.percent‚16.714000701904297 cpu‚26.176270133671284 disk.nvme0c0n1.in‚ 18047.3125 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0$ memory_percent‚97.18234539031982( proc.memory.availableMB‚ 461.64453125 network.sent‚ 2.0314371e+07 network.recv‚ 2.944867e+06) disk./.usagePercent‚39.28394933320073 proc.cpu.threads‚17' disk.nvme0c0n1.out‚29121.3505859375 disk.nvme0n1.in‚ 18047.3125% disk.nvme0n1.out‚29121.3505859375 disk.sda1.out‚0! proc.memory.rssMB‚ 2738.421875$ disk./.usageGB‚585.3592338562012 disk.rbd1.out‚ 37.6416015625,øž´(‚(:“ ¬”ìÒÅÖ• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚272.035' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.407000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sžRúðµ(‚(:å ¬”ìÒöü¾— disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 2.0321459e+07 disk.sda3.out‚ 326.18359375 disk.rbd1.out‚ 37.6416015625$ memory_percent‚97.18081951141357! proc.memory.rssMB‚ 2738.421875( proc.memory.availableMB‚ 461.89453125 disk.sda.out‚ 326.18359375 network.recv‚ 2.948115e+06 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 18047.3125 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 cpu‚26.176040378648413 disk.dm-0.in‚ 18047.3125( disk.nvme0c0n1.out‚29123.77099609375 disk.rbd0.in‚0* proc.memory.percent‚16.714000701904297* disk./.usagePercent‚39.283829777682236 disk.dm-1.out‚ 326.18359375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 18047.3125 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚585.3574523925781 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚29123.77099609375 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚29123.77099609375®tÁ4¶(/ _wandbruntime‚10942  _runtime‚10942U…éÌ·(Æ trainer/global_step‚12749 % _timestamp‚1.7843512839967675e+09 _runtime‚10942.690397513 _step‚1047 ' lr-AdamW/pg1‚0.00013533046202434702 ' lr-AdamW/pg2‚0.00013533046202434702‡×}õѸ(Ë trainer/global_step‚12749 % _timestamp‚1.7843512839967675e+09 _runtime‚10942.690397513 _step‚1047 ' lr-AdamW/pg1‚0.00013533046202434702 ' lr-AdamW/pg2‚0.00013533046202434702—I'h=4¹(/ _wandbruntime‚10942  _runtime‚10942*ðO((º(€P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114603 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49729001 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13478014687744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22332751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1927214892781568 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1937175517913088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49729296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27396253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13457106284301 _step‚1048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1950653549640192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13477673348608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1937175534952448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚12749 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950653191261696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1940672356655119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101052506 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118062097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101052506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27396545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950653191261696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22332748 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426833925203968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1937175517913088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13478014687744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118062097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13477673348608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101052508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440457322526720 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101052508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚10942.694352097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118072074 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426833906591744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118072074 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1927214909820928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13623517094912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937175534952448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0·ˆvŽ»(…P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950653191261696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22332748 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426833925203968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1937175517913088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13478014687744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118062097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13477673348608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101052508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440457322526720 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101052508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚10942.694352097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118072074 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426833906591744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118072074 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1927214909820928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13623517094912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937175534952448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114603 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49729001 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13478014687744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22332751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1927214892781568 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1937175517913088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49729296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27396253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13457106284301 _step‚1048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1950653549640192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13477673348608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1937175534952448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.¬ 9} allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚12749 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950653191261696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1940672356655119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101052506 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118062097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101052506 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27396545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976˜¥’4¼(/ _wandbruntime‚10943  _runtime‚10943½RŠè'½(‹N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13458457888025 _runtime‚10943.308774183 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118081140 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1940819878380825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101060140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49732798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219141280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13478685975552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚12749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950801803303936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440491887305216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426867335350272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101060138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27398588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13624434287104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13479027321856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27398305 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22334496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118071149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1937322775982080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843512846148787e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950801803303936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49733084 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118081140 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950801444918272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1937322758942720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22334493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219131287 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131287 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1937322758942720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚426867353962496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13624533342720 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1927361420492800 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13479027321856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101060140 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13458117338023 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚1049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101060138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927361403453440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283¦>Ÿt'¾(N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101060138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27398588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13624434287104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13479027321856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27398305 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22334496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118071149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1937322775982080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843512846148787e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950801803303936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49733084 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118081140 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950801444918272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1937322758942720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22334493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219131287 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131287 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1937322758942720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚426867353962496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13624533342720 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1927361420492800 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13479027321856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101060140 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13458117338023 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚1049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101060138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927361403453440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13458457888025 _runtime‚10943.308774183 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118081140 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1940819878380825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101060140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49732798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219141280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13478685975552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚12749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950801803303936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440491887305216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426867335350272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664™è· 4¿(/ _wandbruntime‚10943  _runtime‚10943¸ùRÀ(Š 0loss/FBt1/detection_loss‚0.38900054256702143 /loss/FBt2/detection_loss‚0.4225261194358007 % _timestamp‚1.7843512846178174e+09 0loss/FRt1/assignment_loss‚1.4222306788638654 0loss/FRt2/assignment_loss‚0.6284678689504074 epoch‚31 1loss/SRqqt2/detection_loss‚0.6265299225950407 %loss/total_loss‚7.148587940660542 _runtime‚10943.31114057 /loss/FRt1/detection_loss‚0.5480946460106124 1loss/SRqqt1/detection_loss‚0.7802174612198637 trainer/global_step‚12749 _step‚1050 2loss/SRqqt1/assignment_loss‚0.4711831961354689 3loss/SRqqt2/assignment_loss‚0.46657620385601783 3loss/FBt1/assignment_loss‚0.0004294719198170579 4loss/FBt2/assignment_loss‚3.5748110013129815e-05 .loss/FRt2/detection_loss‚1.393296080996614à_ Á( 2loss/SRqqt1/assignment_loss‚0.4711831961354689 3loss/SRqqt2/assignment_loss‚0.46657620385601783 3loss/FBt1/assignment_loss‚0.0004294719198170579 4loss/FBt2/assignment_loss‚3.5748110013129815e-05 .loss/FRt2/detection_loss‚1.393296080996614 0loss/FBt1/detection_loss‚0.38900054256702143 /loss/FBt2/detection_loss‚0.4225261194358007 % _timestamp‚1.7843512846178174e+09 0loss/FRt1/assignment_loss‚1.4222306788638654 0loss/FRt2/assignment_loss‚0.6284678689504074 epoch‚31 1loss/SRqqt2/detection_loss‚0.6265299225950407 %loss/total_loss‚7.148587940660542 _runtime‚10943.31114057 /loss/FRt1/detection_loss‚0.5480946460106124 1loss/SRqqt1/detection_loss‚0.7802174612198637 trainer/global_step‚12749 _step‚1050š.Jª‘“Â(‚(:ˆ »”ìÒˆåÄ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚248.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”a‘éÃ(‚(:Þ »”ìÒ’Á•– disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚29129.0048828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125$ proc.memory.availableMB‚461.5625' disk.nvme0c0n1.out‚29129.0048828125 disk.nvme0n1.in‚ 18047.3125% disk.nvme0n1.out‚29129.0048828125$ memory_percent‚97.18284606933594* proc.memory.percent‚16.714000701904297* disk./.usagePercent‚39.283908628002784 disk.dm-0.in‚ 18047.3125 disk.sda.out‚ 326.51953125 network.sent‚ 2.0324381e+07 disk.sda1.in‚0 network.recv‚ 2.951537e+06 disk.rbd0.in‚0 disk.rbd1.in‚ 2.01953125 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 326.51953125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 18047.3125! proc.memory.rssMB‚ 2738.421875 cpu‚26.175738187608374$ disk./.usageGB‚585.3586273193359 disk.dm-1.out‚ 326.51953125 disk.drbd2165.out‚0 disk.rbd1.out‚ 37.7041015625 proc.cpu.threads‚17¼÷Ú­“Ä(‚(:ˆ Ê”ìÒ©“¶“ gpu.0.gpu‚92.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ón ñîÅ(‚(:ã Ê”ìÒÓã–* proc.memory.percent‚16.714000701904297 cpu‚26.17549910445041 proc.cpu.threads‚17 network.recv‚ 2.955711e+06$ disk./.usageGB‚585.3586578369141 disk.drbd2165.in‚0# disk.dm-0.out‚29131.91064453125& disk.nvme0n1.out‚29131.91064453125 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 18113.7109375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29131.91064453125( proc.memory.availableMB‚ 461.62890625! proc.memory.rssMB‚ 2738.421875 disk.dm-1.out‚ 326.71875 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.18244075775146 disk.sda.out‚ 326.71875 disk.dm-0.in‚ 18113.7109375 disk.sda3.out‚ 326.71875 disk.rbd1.out‚ 37.7041015625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125) disk./.usagePercent‚39.28391067606306! disk.nvme0n1.in‚ 18113.7109375 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.0387522e+07 disk.sda.in‚ 0.1328125 disk.sda1.out‚0½n2žÆ(‚(:“ Ù”ìÒ³™ê“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìïqsñÇ(‚(:æ Ù”ìÒ¡¾…–( disk.nvme0c0n1.out‚29163.49853515625# disk.dm-0.out‚29163.49853515625 disk.sda.out‚ 326.9453125 disk.sda3.in‚ 0.1328125 network.sent‚ 2.0392539e+07 network.recv‚ 2.959463e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 18113.7109375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 18113.7109375% proc.memory.availableMB‚ 462.09375 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.dm-1.out‚ 326.9453125 disk.drbd2165.out‚0 disk.sda3.out‚ 326.9453125 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚16.714000701904297 cpu‚26.175464204589655 proc.cpu.threads‚17& disk.nvme0n1.out‚29163.49853515625 disk.rbd1.out‚ 37.7666015625$ disk./.usageGB‚586.1293716430664 disk.sda2.out‚0! proc.memory.rssMB‚ 2738.421875$ memory_percent‚97.17960357666016( disk./.usagePercent‚39.3356339262659 disk.rbd0.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18113.7109375?˜?4È(/ _wandbruntime‚10983  _runtime‚10983=ËÉ(Å ' lr-AdamW/pg1‚0.00013368194932831805 ' lr-AdamW/pg2‚0.00013368194932831805 trainer/global_step‚12799 $ _timestamp‚1.784351324728771e+09 _runtime‚10983.422407589 _step‚1051ßáÁÐÊ(Ê ' lr-AdamW/pg1‚0.00013368194932831805 ' lr-AdamW/pg2‚0.00013368194932831805 trainer/global_step‚12799 $ _timestamp‚1.784351324728771e+09 _runtime‚10983.422407589 _step‚1051›ËŠ"º4Ë(/ _wandbruntime‚10983  _runtime‚10983äAg˜Â Ì(P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101434106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442179870913024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27499638 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1958066296623616 % _timestamp‚1.7843513247297418e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13528368850432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1948048149146377 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚12799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118526526 G>DeviceStatsMonitor.on_train_batch_start/alÕ÷—Elocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1944537586434048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚1052 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13507722771209 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101434108 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944537569394688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958065938245120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13528710189568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118526526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944537586434048 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1934540443414528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1934540426375168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428505344518144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13528710189568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958065938245120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¹± t (Í(†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22420001 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13674423399936 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1948048506735627 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚12799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118526526 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1944537586434048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚1052 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13507722771209 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101434108 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944537569394688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958065938245120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13528710189568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118526526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944537586434048 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1934540443414528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1934540426375168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428505344518144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13528710189568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958065938245120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101434106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442179870913024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27499638 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1958066296623616 % _timestamp‚1.7843513247297418e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13528368850432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1948048149146377 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1934686937047040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118535592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49923465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13529381477376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13509074374933 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101441738 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219967339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944684827463680 trainer/global_step‚12799 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958214191901696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49923192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1948196028461333 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1944684810424320 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1934686954086400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10984.042329014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977332 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219967339 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27501719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13675434973184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1958214550287360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442214208249856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1053 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118525601 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1958214550287360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101441738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22421746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118535592 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1944684810424320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27501449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944684827463680°±jtGÐ(N WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118535592 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1944684810424320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27501449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944684827463680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13529722823680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428538791888896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13508733824931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13529381477376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1948195670871971 % _timestamp‚1.7843513253484397e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118525601 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1934686937047040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118535592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49923465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13529381477376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13509074374933 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101441738 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219967339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944684827463680 trainer/global_step‚12799 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958214191901696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49923192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1948196028461333 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1944684810424320 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1934686954086400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.larP_boÏge_pool.peak‚1169 _runtime‚10984.042329014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977332 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219967339 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27501719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13675434973184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1958214550287360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442214208249856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1053 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118525601 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1958214550287360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101441738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22421746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983qúð4Ñ(/ _wandbruntime‚10984  _runtime‚10984²&ÈÒ(Š 0loss/FRt1/assignment_loss‚1.4829352086690921 4loss/FBt1/assignment_loss‚5.3657860503067115e-05 1loss/SRqqt1/detection_loss‚0.7411707870324038 1loss/SRqqt2/detection_loss‚0.6324517429612081 /loss/FBt1/detection_loss‚0.5511488576626612 0loss/FBt2/detection_loss‚0.45612612776413486 % _timestamp‚1.7843513253522642e+09 _runtime‚10984.045582463 %loss/total_loss‚7.476550545206357 _step‚1054 1loss/SRqqt1/assignment_loss‚0.516891167416361 3loss/SRqqt2/assignment_loss‚0.42328604262983743 /loss/FRt1/detection_loss‚0.5266467181758814 0loss/FRt2/assignment_loss‚0.6164595164278502 2loss/FBt2/assignment_loss‚0.008022571445855019 /loss/FRt2/detection_loss‚1.5213581471605682 epoch‚31 trainer/global_step‚12799­ÂíÓ( 0loss/FRt2/assignment_loss‚0.6164595164278502 2loss/FBt2/assignment_loss‚0.008022571445855019 /loss/FRt2/detection_loss‚1.5213581471605682 epoch‚31 trainer/global_step‚12799 0loss/FRt1/assignment_loss‚1.4829352086690921 4loss/FBt1/assignment_loss‚5.3657860503067115e-05 1loss/SRqqt1/detection_loss‚0.7411707870324038 1loss/SRqqt2/detection_loss‚0.6324517429612081 /loss/FBt1/detection_loss‚0.5511488576626612 0loss/FBt2/detection_loss‚0.45612612776413486 % _timestamp‚1.7843513253522642e+09 _runtime‚10984.045582463 %loss/total_loss‚7.476550545206357 _step‚1054 1loss/SRqqt1/assignment_loss‚0.516891167416361 3loss/SRqqt2/assignment_loss‚0.42328604262983743 /loss/FRt1/detection_loss‚0.5266467181758814ž';—‚‘Ô(‚(:† è”ìÒ›µó– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.474' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÐp¾ìÕ(‚(:á è”ìÒ›´ð˜$ disk./.usageGB‚586.1007118225098# proc.memory.rssMB‚ 2752.71484375 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚327.3125( disk.nvme0c0n1.out‚29166.44287109375 network.recv‚ 2.962814e+06 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.33371054166115 disk.sda2.out‚0& disk.nvme0n1.out‚29166.44287109375 disk.sda3.out‚327.3125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 18113.7109375) proc.memory.percent‚16.80123805999756 network.sent‚ 2.0397408e+07 disk.rbd1.out‚ 37.7666015625 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.27180004119873( proc.memory.availableMB‚ 446.98828125 disk.drbd2165.in‚0 cpu‚26.17775135996847# disk.dm-0.out‚29166.44287109375 disk.sda.out‚327.3125 disk.sda2.in‚0! disk.nvme0n1.in‚ 18113.7109375 disk.rbd0.in‚0 disk.dm-0.in‚ 18113.7109375¶v(4Ö(/ _wandbruntime‚11002  _runtime‚11002qD½ce=×(ßz 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6959694027900696 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025548560544848442 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971825480461121 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2394321709871292 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.655065655708313 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6096358895301819 _step‚1055 )loss/val_total_loss‚7.670980670687602 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1368296891450882 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2152475118637085 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23913617432117462 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7420651316642761 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_1_of_5‚1 trainer/global_step‚12799 % _timestamp‚1.7843513440560787e+09 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24692969024181366 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7660754919052124 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6294398307800293 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.847449004650116 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018155410885810852 _runtime‚11002.750331397 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23491975665092468 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6312494874000549 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3663947284221649 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2859209179878235 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05921228229999542 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28680047392845154 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027838729321956635 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6974445581436157 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6783804297447205 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4891081750392914 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7593095898628235 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08186393231153488 +jet/accuracy_4_of_4‚0.01908157207071781 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8593348264694214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24785427749156952 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18057332932949066 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34073448181152344 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7744863629341125 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2798575460910797 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2324848175048828 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3167555332183838 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7800539135932922 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2856423258781433 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7392854690551758 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12830007076263428 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6404250264167786 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8723111152648926 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6451020240783691 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954627156257629 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6602684259414673 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12065814435482025 particle/accuracy_4_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6404250264167786 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9943978190422058 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7922011017799377 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980711936950684 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9504482746124268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29659485816955566 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3816456198692322 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.677741289138794 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7271963953971863 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 epoch‚31 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2150340974330902 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03530413284897804 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958568811416626 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.712297797203064 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3778902590274811 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932328462600708 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6734116077423096 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6362629532814026 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6172531247138977 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7399249076843262 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7260943651199341 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1572912186384201 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3326200246810913 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8877690434455872 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5664117336273193 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8009636998176575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8031548857688904 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2226346731185913 /particle/accuracy_4_of_4‚0.9436118006706238 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15337519347667694 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891518354415894 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12872301042079926 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9000377655029297 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025359561666846275 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8341471552848816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27711206674575806 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8723111152648926 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23513853549957275 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018155410885810852 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6808282732963562 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23661457002162933 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.34872329235076904 +particle/specificity‚0.8973473310470581 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7462695240974426 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0943637490272522 *jet/accuracy_1_of_4‚0.9852102398872375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211093902587891 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027559485286474228 *jet/accuracy_1_of_3‚0.9262795448303223 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5847357511520386 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04847263544797897 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29695117473602295 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994893491268158 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5617151260375977 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5377362370491028 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29856324195861816 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.249914288520813 *jet/accuracy_2_of_4‚0.7485780119895935 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7855575084686279 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6408774256706238 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13685369491577148 +jet/accuracy_3_of_3‚0.04297542944550514 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04212413728237152 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_3‚0.5098230242729187 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2859209179878235 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03619268164038658 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7105687856674194 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895318150520325 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8040315508842468 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22692732512950897 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5221074819564819 *jet/accuracy_1_of_2‚0.7884997129440308 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8781178593635559 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8840702772140503 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7751886248588562 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48228418827056885 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7429906129837036 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.03828353062272072 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6534824371337891 +particle/sensitivity‚0.6030580997467041 ("Purity/2FRt1SRqqt1FBt/event_purity‚0úo§ƒ%Ø(äz 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9943978190422058 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7922011017799377 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980711936950684 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9504482746124268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29659485816955566 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3816456198692322 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.677741289138794 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7271963953971863 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 epoch‚31 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2150340974330902 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03530413284897804 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958568811416626 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.712297797203064 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3778902590274811 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9932328462600708 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6734116077423096 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6362629532814026 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6172531247138977 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7399249076843262 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7260943651199341 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1572912186384201 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3326200246810913 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8877690434455872 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5664117336273193 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8009636998176575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8031548857688904 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2226346731185913 /particle/accuracy_4_of_4‚0.9436118006706238 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15337519347667694 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9891518354415894 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12872301042079926 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9000377655029297 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025359561666846275 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8341471552848816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27711206674575806 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8723111152648926 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23513853549957275 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018155410885810852 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6808282732963562 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23661457002162933 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.34872329235076904 +particle/specificity‚0.8973473310470581 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7462695240974426 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0943637490272522 *jet/accuracy_1_of_4‚0.9852102398872375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7211093902587891 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027559485286474228 *jet/accuracy_1_of_3‚0.9262795448303223 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5847357511520386 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04847263544797897 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29695117473602295 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994893491268158 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5617151260375977 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5377362370491028 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29856324195861816 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.249914288520813 *jet/accuracy_2_of_4‚0.7485780119895935 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7855575084686279 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6408774256706238 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13685369491577148 +jet/accuracy_3_of_3‚0.04297542944550514 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04212413728237152 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_3‚0.5098230242729187 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2859209179878235 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03619268164038658 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7105687856674194 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895318150520325 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8040315508842468 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22692732512950897 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5221074819564819 *jet/accuracy_1_of_2‚0.7884997129440308 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8781178593635559 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8840702772140503 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7751886248588562 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48228418827056885 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 A&Purity/1FRt2SRqqt1F‚E'çBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7429906129837036 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.03828353062272072 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6534824371337891 +particle/sensitivity‚0.6030580997467041 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6959694027900696 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025548560544848442 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971825480461121 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2394321709871292 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.655065655708313 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6096358895301819 _step‚1055 )loss/val_total_loss‚7.670980670687602 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1368296891450882 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2152475118637085 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23913617432117462 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7420651316642761 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_1_of_5‚1 trainer/global_step‚12799 % _timestamp‚1.7843513440560787e+09 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24692969024181366 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7660754919052124 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6294398307800293 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.847449004650116 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018155410885810852 _runtime‚11002.750331397 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23491975665092468 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6312494874000549 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3663947284221649 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2859209179878235 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05921228229999542 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28680047392845154 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027838729321956635 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6974445581436157 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6783804297447205 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4891081750392914 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7593095898628235 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08186393231153488 +jet/accuracy_4_of_4‚0.01908157207071781 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8593348264694214 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24785427749156952 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18057332932949066 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34073448181152344 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7744863629341125 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2798575460910797 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2324848175048828 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3167555332183838 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7800539135932922 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2856423258781433 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7392854690551758 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12830007076263428 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6404250264167786 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8723111152648926 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6451020240783691 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954627156257629 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6602684259414673 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12065814435482025 particle/accuracy_4_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6404250264167786 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt0SRqqt2FBt/event_purity‚1Ÿ"/Ú“Ù(‚(:ˆ ÷”ìҺЇ” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í–N®óÚ(‚(:è ÷”ìÒÔò²– network.recv‚ 2.967452e+06 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 18113.7109375 disk.rbd1.in‚ 2.01953125) disk./.usagePercent‚39.33363271537074 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 436.42578125" proc.memory.rssMB‚ 2763.2578125 disk.dm-1.out‚ 327.75390625" disk.dm-0.out‚29958.3271484375 network.sent‚ 2.0460542e+07 disk.sda2.out‚0) proc.memory.percent‚16.86558723449707' disk.nvme0c0n1.out‚29958.3271484375 disk.dm-0.in‚ 18113.7109375 disk.sda2.in‚0 disk.rbd1.out‚ 37.8916015625# disk./.usageGB‚586.099552154541 disk.sda1.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚29958.3271484375 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 18113.7109375 disk.sda.out‚ 327.75390625 disk.sda3.out‚ 327.75390625# memory_percent‚97.3362684249878 cpu‚26.179402012220642Ú©v¯žÛ(‚(:“ †•ì҈˪“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚257.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.492399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TCéEæÜ(‚(:Û †•ìÒ äÚ– disk.rbd1.out‚ 37.8916015625 network.recv‚ 2.97149e+06 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 cpu‚26.179315249984946 disk.sda3.in‚ 0.1328125$ memory_percent‚97.33712673187256 disk.sda3.out‚328.25 disk.rbd0.in‚0 disk.dm-1.out‚328.25 disk.drbd2165.out‚0 disk.dm-0.in‚18175.71484375 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 436.28515625" disk.nvme0n1.in‚18175.71484375$ disk.nvme0c0n1.in‚18175.71484375' disk.nvme0c0n1.out‚29962.5576171875" disk.dm-0.out‚29962.5576171875 disk.sda.out‚328.25) proc.memory.percent‚16.86558723449707 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 2.0517758e+07* disk./.usagePercent‚39.283987990338396$ disk./.usageGB‚585.3598098754883% disk.nvme0n1.out‚29962.5576171875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2763.2578125¢+’“Ý(‚(:ˆ ••ìÒô€¯“ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÔÒ+õÞ(‚(:ê ••ìÒæÈÖ• disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda3.in‚ 0.1328125* disk./.usagePercent‚39.282191585470635 disk.sda1.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚29969.47802734375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 328.609375 disk.rbd1.out‚ 37.9541015625 network.sent‚ 2.0522743e+07& disk.nvme0n1.out‚29969.47802734375 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 2.974954e+06$ disk./.usageGB‚585.3330421447754$ disk.nvme0c0n1.in‚18175.71484375 disk.dm-0.in‚18175.71484375" proc.memory.rssMB‚ 2763.2578125) proc.memory.percent‚16.86558723449707 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚29969.47802734375" disk.nvme0n1.in‚18175.71484375 cpu‚26.179167636241026 disk.sda3.out‚ 328.609375$ memory_percent‚97.33614921569824' proc.memory.availableMB‚ 436.4453125 proc.cpu.threads‚17 disk.dm-1.out‚ 328.609375 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125Í#Õ4ß(/ _wandbruntime‚11043  _runtime‚11043i“Ìà(Æ _runtime‚11043.291523186 _step‚1056 ' lr-AdamW/pg1‚0.00013203927217691794 ' lr-AdamW/pg2‚0.00013203927217691794 trainer/global_step‚12849 % _timestamp‚1.7843513845948331e+09œ‘dÑá(Ë % _timestamp‚1.7843513845948331e+09 _runtime‚11043.291523186 _step‚1056 ' lr-AdamW/pg1‚0.00013203927217691794 ' lr-AdamW/pg2‚0.00013203927217691794 trainer/global_step‚12849 @“cX4â(/ _wandbruntime‚11043  _runtime‚11043`sô(ã(€P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443936320053760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220977695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27633312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13743625509376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1953829204566016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13596797557760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101936372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119051300 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50143083 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1957350682166194 $ _timestamp‚1.784351384598769e+09 _step‚1057 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1967425985084416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50142797 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22509771 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119041323 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27633029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430192713156608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13596797557760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101936374 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1967426343462912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13743722475008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22509768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101936372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13597138896896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1953829187526656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443936435631616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13597138896896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1953829204566016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13596797557760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101936372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119051300 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50143083 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1957350682166194 $ _timestamp‚1.784351384598769e+09 _step‚1057 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1967425985084416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50142797 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22509771 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119041323 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27633029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430192713156608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13596797557760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101936374 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1967426343462912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13743722475008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22509768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101936372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13597138896896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1953829187526656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443936435631616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13597138896896 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50146912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1058 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220994379 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101944006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requesteÁ3^A>"d_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1967574597126656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101944004 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1943921133854720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13577410587070 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220994379 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221004372 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1967574597126656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27635396 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1953976428556288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1953976445595648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1957498561481150 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12849 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119050375 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13577070037068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430226123302912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1953976428556288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119050375 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1953976445595648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101944006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚11043.909252397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119060366 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22511513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119060366 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13597810184704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50146619 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12849 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119050375 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13577070037068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430226123302912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1953976428556288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119050375 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1953976445595648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101944006 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚11043.909252397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119060366 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22511513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119060366 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13597810184704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50146619 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¢úñ-ð4è(/ _wandbruntime‚11043  _runtime‚11043=E÷Ž é(‡ 2loss/SRqqt2/assignment_loss‚0.4349914605734144 1loss/SRqqt1/detection_loss‚0.7393445037605321 /loss/FBt1/detection_loss‚0.4223376233291266 %loss/total_loss‚6.870384882341558 _runtime‚ 11043.9121338 0loss/FRt1/assignment_loss‚1.4539583630135997 .loss/FRt2/detection_loss‚1.318026214941784 trainer/global_step‚12849 3loss/FBt1/assignment_loss‚0.0002744164635558867 3loss/FBt2/assignment_loss‚0.0015831432239972842 1loss/SRqqt2/detection_loss‚0.5742548659616815 _step‚1059 0loss/FRt2/assignment_loss‚0.6068468555204396 2loss/SRqqt1/assignment_loss‚0.4730757492671711 /loss/FRt1/detection_loss‚0.5409938798385799 0loss/FBt2/detection_loss‚0.30469780644767563 epoch‚32 % _timestamp‚1.7843513852188404e+09åÀ ê(Œ 2loss/SRqqt2/assignment_loss‚0.4349914605734144 1loss/SRqqt1/detection_loss‚0.7393445037605321 /loss/FBt1/detection_loss‚0.4223376233291266 %loss/total_loss‚6.870384882341558 _runtime‚ 11043.9121338 0loss/FRt1/assignment_loss‚1.4539583630135997 .loss/FRt2/detection_loss‚1.318026214941784 trainer/global_step‚12849 3loss/FBt1/assignment_loss‚0.0002744164635558867 3loss/FBt2/assignment_loss‚0.0015831432239972842 1loss/SRqqt2/detection_loss‚0.5742548659616815 _step‚1059 0loss/FRt2/assignment_loss‚0.6068468555204396 2loss/SRqqt1/assignment_loss‚0.4730757492671711 /loss/FRt1/detection_loss‚0.5409938798385799 0loss/FBt2/detection_loss‚0.30469780644767563 epoch‚32 % _timestamp‚1.7843513852188404e+09£ºL½2ë(‚(:’ ¤•ìÒ‹›Ð” gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûØ?öì(‚(:ë ¤•ìÒßçÿ• disk.sda1.out‚0 disk.sda3.out‚ 328.796875( proc.memory.availableMB‚ 436.45703125$ memory_percent‚97.33607769012451$ disk.nvme0c0n1.in‚18175.71484375 network.recv‚ 2.978549e+06* disk./.usagePercent‚39.281716947502154 disk.dm-1.out‚ 328.796875 disk.sda2.out‚0 cpu‚26.178810791666017 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚29971.93896484375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚18175.71484375 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 network.sent‚ 2.0527825e+07 disk.drbd2165.out‚0 disk.sda.out‚ 328.796875" disk.nvme0n1.in‚18175.71484375$ disk./.usageGB‚585.3259696960449 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.86558723449707 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚29971.93896484375 proc.cpu.threads‚17 disk.rbd0.in‚0# disk.dm-0.out‚29971.93896484375 disk.rbd1.out‚ 37.9541015625" proc.memory.rssMB‚ 2763.2578125L·Ä“í(‚(:ˆ ³•ìÒ­’„• gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Pbñôî(‚(:é ³•ìÒ‘ü– proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚29977.69580078125 disk.rbd1.out‚ 38.0166015625 disk.dm-1.out‚ 329.234375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚18175.71484375 disk.sda3.out‚ 329.234375) proc.memory.percent‚16.86558723449707 cpu‚26.17865664022302) disk./.usagePercent‚39.28171566746448 network.sent‚ 2.0590795e+07 disk.rbd0.out‚0" disk.nvme0n1.in‚18175.71484375& disk.nvme0n1.out‚29977.69580078125$ disk./.usageGB‚585.3259506225586 disk.sda1.out‚0( disk.nvme0c0n1.out‚29977.69580078125 disk.sda.out‚ 329.234375$ memory_percent‚97.33574390411377 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 436.51171875 network.recv‚ 2.983362e+06$ disk.nvme0c0n1.in‚18175.71484375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125" proc.memory.rssMB‚ 2763.25781257ÈÒU4ï(/ _wandbruntime‚11084  _runtime‚11084ÄøÜâËð(Å ' lr-AdamW/pg1‚0.00013040253607623045 ' lr-AdamW/pg2‚0.00013040253607623045 trainer/global_step‚12899 $ _timestamp‚1.784351425379804e+09 _runtime‚11084.076487121 _step‚1060R³©£Ðñ(Ê trainer/global_step‚12899 $ _timestamp‚1.784351425379804e+09 _runtime‚11084.076487121 _step‚1060 ' lr-AdamW/pg1‚0.00013040253607623045 ' lr-AdamW/pg2‚0.00013040253607623045¤L{Có4ò(/ _wandbruntime‚11084  _runtime‚11084šUÎ`Ù"ó(P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22597018 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221813747 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431864151083008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13626675470254 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221823726 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27737280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50334301 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50334046 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚11084.081043149 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13647834398720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119495775 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1974838732067840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22597021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13794913870336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1964726832246702 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1061 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119495775 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1961191239008256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102317974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445659064953344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431864132470784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102317974 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1961191256047616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1974838732067840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627016020144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119505752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119505752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27737028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚12899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964727189835952 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1951100156776448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102317974 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1961191256047616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1974838732067840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627016020144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119505752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119505752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27737028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚12899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964727189835952 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1951100156776448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221813747 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431864151083008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13626675470254 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221823726 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27737280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50334301 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50334046 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚11084.081043149 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13647834398720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119495775 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1974838732067840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22597021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13794913870336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1964726832246702¥v«hè4õ(/ _wandbruntime‚11084  _runtime‚11084p]Ôï'ö(‹N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13648847032832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50337776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445693396559872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119514818 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119514818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22598763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102325606 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1964874353972296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1961338480037888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431897579841536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13795835330560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1974986985724416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13648505686528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431897561229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1974987344110080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚11084.882484016 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1974987344110080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13628027073978 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843514261886947e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27739013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664oÉFm'÷(N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119504827 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102325604 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 trainer/global_step‚12899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221830431 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13648847032832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50337776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445693396559872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119514818 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119514818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22598763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102325606 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1964874353972296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1961338480037888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431897579841536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13795835330560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1974986985724416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13648505686528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431897561229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1974987344110080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚11084.882484016 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1974987344110080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13628027073978 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843514261886947e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27739013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7843514663434143e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1958425690370048 trainer/global_step‚12949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649799 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102699574 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1968553307529216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1065 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13698529900544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13677632507052 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13698188561408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22684268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1972103339916460 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1968553290489856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119950227 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1982251837429760 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982251479051264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982251479051264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102699574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13845871070720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13677291957162 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102699572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50525158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13698188561408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1982251837429760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27840633 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1972102982327210 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269þp¾6 (†)†P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982251479051264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102699574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13845871070720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13677291957162 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102699572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50525158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13698188561408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1982251837429760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27840633 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1972102982327210 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102699572 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50524901 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7843514663434143e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1958425690370048 trainer/global_step‚12949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649799 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102699574 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1968553307529216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1065 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13698529900544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13677632507052 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13698188561408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22684268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1972103339916460 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1968553290489856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119950227 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1982251837429760 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982251479051264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0©àb U4‡)/ _wandbruntime‚11125  _runtime‚11125«ê¶‹ˆ)ˆN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222666483 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222676476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1982400091093504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1968700531519488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13678643560886 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13678303010884 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13846983079936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27842676 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13699201188352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13699542534656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1972250504052804 _step‚1066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433568999155712 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1982400091093504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13699542534656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1982399732707840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676476 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119969270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102707204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22686013 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119969270 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1968700548558848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222666483 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13846890315776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22686016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102707206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27842938 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50528689 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447415889471488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102707204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447416000847872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119959279 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚12949 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 " _timestamp‚1.7843514669599e+09 _runtime‚11125.653752964 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13699201188352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50528954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1958572218081280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1958572201041920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438m0>ž'‰)N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222666483 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222676476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1982400091093504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1968700531519488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13678643560886 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13678303010884 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13846983079936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27842676 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13699201188352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13699542534656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1972250504052804 _step‚1066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433568999155712 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1982400091093504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13699542534656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1982399732707840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676476 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119969270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102707204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22686013 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119969270 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1968700548558848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222666483 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13846890315776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22686016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102707206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27842938 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50528689 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447415889471488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102707204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447416000847872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119959279 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚12949 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 " _timestamp‚1.7843514669599e+09 _runtime‚11125.653752964 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13699201188352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50528954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1958572218081280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1958572201041920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ª`Ðlæ4Š)/ _wandbruntime‚11125  _runtime‚11125 î1|‹)‰ /loss/FRt1/detection_loss‚0.5210831418182118 /loss/FRt2/detection_loss‚1.4284300907676313 /loss/FBt2/detection_loss‚0.5521261377217012 $ _timestamp‚1.784351466963658e+09 _step‚1067 /loss/FRt1/assignment_loss‚1.486452695224557 1loss/SRqqt2/detection_loss‚0.6093965634709728 0loss/FBt1/detection_loss‚0.24269832706444763 trainer/global_step‚12949 3loss/SRqqt1/assignment_loss‚0.46000515918459217 2loss/SRqqt2/assignment_loss‚0.4190955133526715 1loss/SRqqt1/detection_loss‚0.7527361520034962 &loss/total_loss‚7.0893105977942605 epoch‚32 _runtime‚11125.656971586 0loss/FRt2/assignment_loss‚0.6029811593765708 4loss/FBt1/assignment_loss‚0.00013852056402592034 1loss/FBt2/assignment_loss‚0.01416713724538264³ÞÚÇŒ)Ž 1loss/SRqqt1/detection_loss‚0.7527361520034962 &loss/total_loss‚7.0893105977942605 epoch‚32 _runtime‚11125.656971586 0loss/FRt2/assignment_loss‚0.6029811593765708 4loss/FBt1/assignment_loss‚0.00013852056402592034 1loss/FBt2/assignment_loss‚0.01416713724538264 /loss/FRt1/detection_loss‚0.5210831418182118 /loss/FRt2/detection_loss‚1.4284300907676313 /loss/FBt2/detection_loss‚0.5521261377217012 $ _timestamp‚1.784351466963658e+09 _step‚1067 /loss/FRt1/assignment_loss‚1.486452695224557 1loss/SRqqt2/detection_loss‚0.6093965634709728 0loss/FBt1/detection_loss‚0.24269832706444763 trainer/global_step‚12949 3loss/SRqqt1/assignment_loss‚0.46000515918459217 2loss/SRqqt2/assignment_loss‚0.4190955133526715«Ùïc,)‚(:’ ï•ìÒåß™” gpu.0.gpu‚13.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ž öóŽ)‚(:è ï•ìҜͱ• disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 330.36328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 38.1416015625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 18245.0390625$ disk./.usageGB‚585.3267860412598 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 330.36328125 disk.dm-0.in‚ 18245.0390625" disk.dm-0.out‚29994.8525390625" proc.memory.rssMB‚ 2763.2578125 cpu‚26.17776879596256 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.0669125e+07 disk.sda1.in‚0' disk.nvme0c0n1.out‚29994.8525390625! disk.nvme0n1.in‚ 18245.0390625% disk.nvme0n1.out‚29994.8525390625$ memory_percent‚97.33614921569824) disk./.usagePercent‚39.28177173311448) proc.memory.percent‚16.86558723449707 disk.sda.out‚ 330.36328125 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125' proc.memory.availableMB‚ 436.4453125 network.recv‚ 2.997869e+06ÃÿàÈ)‚(:… þ•ìÒö¢À“ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_MeCû)‚(:ð þ•ìÒ‡¯Ñ— disk.dm-1.out‚ 330.91796875 cpu‚26.177586288237695 network.sent‚ 2.0732093e+07) disk./.usagePercent‚39.28178555752133$ disk.nvme0c0n1.in‚18311.22265625 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 330.91796875 disk.dm-1.in‚ 0.1328125( proc.memory.availableMB‚ 436.98828125$ disk./.usageGB‚585.3269920349121& disk.nvme0n1.out‚29998.39599609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2763.2578125 disk.sda.out‚ 330.91796875 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.percent‚16.86558723449707 disk.sda1.out‚0$ memory_percent‚97.33283519744873 proc.cpu.threads‚17( disk.nvme0c0n1.out‚29998.39599609375 disk.dm-0.in‚18311.22265625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125# disk.dm-0.out‚29998.39599609375" disk.nvme0n1.in‚18311.22265625 disk.drbd2165.in‚0 network.recv‚ 3.001921e+06 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 38.1416015625šŽL“‘)‚(:ˆ –ìÒŽ¸Å“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D¤LÚô’)‚(:é –ìÒÔÑž– disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 38.2041015625 cpu‚26.177399721979345 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 331.2734375" proc.memory.rssMB‚ 2763.2578125 disk.dm-1.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚30003.3134765625 disk.dm-0.in‚18311.22265625" disk.nvme0n1.in‚18311.22265625 disk.sda3.out‚ 331.2734375 network.recv‚ 3.005488e+06) disk./.usagePercent‚39.28178914162681 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 331.2734375$ memory_percent‚97.33576774597168$ disk.nvme0c0n1.in‚18311.22265625' proc.memory.availableMB‚ 436.5078125$ disk./.usageGB‚585.3270454406738" disk.dm-0.out‚30003.3134765625 disk.sda2.in‚0% disk.nvme0n1.out‚30003.3134765625 disk.sda2.out‚0) proc.memory.percent‚16.86558723449707 network.sent‚ 2.0737685e+07 disk.sda.in‚ 0.1328125ˆ$>4“)/ _wandbruntime‚11166  _runtime‚11166°§¡ÚÌ”)Æ _runtime‚11166.045338366 _step‚1068 ' lr-AdamW/pg1‚0.00012714730713666393 ' lr-AdamW/pg2‚0.00012714730713666393 trainer/global_step‚12999 % _timestamp‚1.7843515073517873e+09þ¥V&Ñ•)Ë _step‚1068 ' lr-AdamW/pg1‚0.00012714730713666393 ' lr-AdamW/pg2‚0.00012714730713666393 trainer/global_step‚12999 % _timestamp‚1.7843515073517873e+09 _runtime‚11166.045338366¬ÕP4–)/ _wandbruntime‚11166  _runtime‚11166@øþ(—)P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1975915359010816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989664584413184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22771521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435207008323584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13896980400128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989664226034688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50715711 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449103904340992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13896896017408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1069 _runtime‚11166.046263893 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1965751223963648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103081174 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50715450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843515073525608e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120404679 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1965751241003008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449104007335936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22771518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103081172 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1979479132407718 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120414656 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1975915341971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223485851 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13748884063232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120414656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚12999 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1975915341971456 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1979479489996968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27943932 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120404679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0²pÕ%î ˜)†P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚12999 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1975915341971456 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1979479489996968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27943932 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120404679 JBDeviceStatsMonitor.on_train_batch_stamWKÌrt/inactive_split.all.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1975915359010816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989664584413184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22771521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435207008323584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13896980400128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989664226034688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50715711 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449103904340992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13896896017408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1069 _runtime‚11166.046263893 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1965751223963648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103081174 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50715450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843515073525608e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120404679 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1965751241003008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449104007335936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22771518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103081172 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1979479132407718 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120414656 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1975915341971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223485851 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13748884063232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120414656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269­2Fp¬4™)/ _wandbruntime‚11166  _runtime‚11166‚—åî'š)‹N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435240455694336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13750238036480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1976062600040448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _runtime‚11166.664711675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22773266 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449138359444992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13729260047794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502535 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435240437082112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13898006738432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1979627011722674 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502535 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1965897751674880 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1976062583001088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1070 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120423722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27946255 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1965897734635520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50719242 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103088806 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1976062600040448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120423722 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50719521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1976062583001088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120413731 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13749896690176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13897922362880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1979626654133312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103088804 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449138462432768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1989812838076928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50719242 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103088806 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1976062600040448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120423722 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50719521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1976062583001088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120413731 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13749896690176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13897922362880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1979626654133312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103088804 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449138462432768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1989812838076928 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13750238036480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1976062600040448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _runtime‚11166.664711675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22773266 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449138359444992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13729260047794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502535 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435240437082112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13898006738432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1979627011722674 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502535 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0®owWø4œ)/ _wandbruntime‚11166  _runtime‚11166ªŽ½: )… 0loss/FRt2/assignment_loss‚0.5989998579025269 1loss/SRqqt1/detection_loss‚0.7561098508279216 _step‚1071 /loss/FRt1/assignment_loss‚1.449510244815522 2loss/SRqqt1/assignment_loss‚0.5270793023495658 2loss/SRqqt2/assignment_loss‚0.4106070273120192 /loss/FRt1/detection_loss‚0.5304683154160416 %loss/total_loss‚7.135139613751104 % _timestamp‚1.7843515079734814e+09 2loss/FBt2/assignment_loss‚0.008939523511519129 1loss/SRqqt2/detection_loss‚0.6013282916024665 0loss/FBt2/detection_loss‚0.32405378511711525 epoch‚32 1loss/FBt1/assignment_loss‚0.00775480408994822 .loss/FRt2/detection_loss‚1.482213931859881 /loss/FBt1/detection_loss‚0.4380746789465775 trainer/global_step‚12999 _runtime‚11166.666796711…W ž)Š .loss/FRt2/detection_loss‚1.482213931859881 /loss/FBt1/detection_loss‚0.4380746789465775 trainer/global_step‚12999 _runtime‚11166.666796711 0loss/FRt2/assignment_loss‚0.5989998579025269 1loss/SRqqt1/detection_loss‚0.7561098508279216 _step‚1071 /loss/FRt1/assignment_loss‚1.449510244815522 2loss/SRqqt1/assignment_loss‚0.5270793023495658 2loss/SRqqt2/assignment_loss‚0.4106070273120192 /loss/FRt1/detection_loss‚0.5304683154160416 %loss/total_loss‚7.135139613751104 % _timestamp‚1.7843515079734814e+09 2loss/FBt2/assignment_loss‚0.008939523511519129 1loss/SRqqt2/detection_loss‚0.6013282916024665 0loss/FBt2/detection_loss‚0.32405378511711525 epoch‚32 1loss/FBt1/assignment_loss‚0.00775480408994822¯.l—•“Ÿ)‚(:ˆ œ–ìÒõ¨” gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ]þô )‚(:é œ–ìÒ¡Íá•" disk.dm-0.out‚30005.8876953125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 331.6640625% disk.nvme0n1.out‚30005.8876953125 disk.rbd1.out‚ 38.2041015625" proc.memory.rssMB‚ 2763.2578125) proc.memory.percent‚16.86558723449707 network.recv‚ 3.007947e+06 disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚18311.22265625 disk.sda1.out‚0) disk./.usagePercent‚39.28178094938571' disk.nvme0c0n1.out‚30005.8876953125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125$ memory_percent‚97.33588695526123 proc.cpu.threads‚17$ disk./.usageGB‚585.3269233703613 disk.dm-1.out‚ 331.6640625 disk.drbd2165.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 436.48828125 disk.rbd0.in‚0 network.sent‚ 2.074123e+07" disk.nvme0n1.in‚18311.22265625 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 cpu‚26.177253776795315 disk.drbd2165.in‚0 disk.sda3.out‚ 331.6640625 disk.sda1.in‚0 disk.dm-0.in‚18311.22265625'Xºøž¡)‚(:“ «–ìÒÚ­Ý” gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?$S#ò¢)‚(:ç «–ìҸǢ–( disk.nvme0c0n1.out‚30009.87451171875$ disk./.usageGB‚585.3252220153809 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚18311.22265625" proc.memory.rssMB‚ 2763.2578125 proc.cpu.threads‚17 network.recv‚ 3.012741e+06 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚30009.87451171875( proc.memory.availableMB‚ 436.55078125 network.sent‚ 2.0804347e+07 disk.rbd1.out‚ 38.2666015625$ memory_percent‚97.33550548553467) disk./.usagePercent‚39.28166677002544 disk.sda3.out‚ 331.84375 disk.rbd0.in‚0" disk.nvme0n1.in‚18311.22265625 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 331.84375 disk.drbd2165.out‚0 disk.sda2.in‚0) proc.memory.percent‚16.86558723449707& disk.nvme0n1.out‚30009.87451171875 disk.dm-0.in‚18311.22265625 disk.sda.in‚ 0.1328125 disk.sda.out‚ 331.84375 disk.sda2.out‚0 cpu‚26.177014312804925C}ï“£)‚(:ˆ º–ìÒº¡Û“ gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.946' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Ú&@3¤)‚(:ß º–ìÒ†‹ˆ–# memory_percent‚97.3348617553711% proc.memory.availableMB‚ 436.65625 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 38.2666015625! disk.nvme0n1.in‚ 18317.0859375 disk.sda3.out‚ 332.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.o§B‹Ú·ut‚ 332.13671875 network.recv‚ 3.016228e+06) disk./.usagePercent‚39.28165934580694 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.dm-0.out‚30013.01171875 network.sent‚ 2.0809223e+07$ disk./.usageGB‚585.3251113891602 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 18317.0859375# disk.nvme0n1.out‚30013.01171875" proc.memory.rssMB‚ 2763.2578125 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0% disk.nvme0c0n1.out‚30013.01171875 disk.dm-0.in‚ 18317.0859375 disk.sda2.out‚0 cpu‚26.17675669312668 proc.cpu.threads‚17 disk.sda.out‚ 332.13671875 disk.sda2.in‚0) proc.memory.percent‚16.86558723449707N?»4¥)/ _wandbruntime‚11206  _runtime‚11206[ü2̦)Æ ' lr-AdamW/pg2‚0.00012552902337505692 trainer/global_step‚13049 % _timestamp‚1.7843515481357658e+09 _runtime‚11206.829356591 _step‚1072 ' lr-AdamW/pg1‚0.00012552902337505692î"[‡Ñ§)Ë trainer/global_step‚13049 % _timestamp‚1.7843515481357658e+09 _runtime‚11206.829356591 _step‚1072 ' lr-AdamW/pg1‚0.00012552902337505692 ' lr-AdamW/pg2‚0.00012552902337505692°ÎÍZ4¨)/ _wandbruntime‚11206  _runtime‚11206:r/(©)P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103462774 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1983277410492416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1983277393453056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1997076973018112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13947921089536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22858768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450826480820224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1986855282488226 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50905816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚13049 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103462772 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚11206.830291927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331882 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1973076774596608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1997077331396608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28047045 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436878446249984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13948015958016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997077331396608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22858771 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50905558 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13799920904192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120869108 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997076973018112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13799579565056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1973076757557248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13778865480868 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13799579565056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843515481365862e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224321903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13799920904192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1986855640077476 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13778524930978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1983277393453056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997076973018112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13799579565056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1973076757557248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13778865480868 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13799579565056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843515481365862e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224321903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13799920904192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1986855640077476 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13778524930978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1983277393453056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚11206.830291927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331882 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1973076774596608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1997077331396608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28047045 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436878446249984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13948015958016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997077331396608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22858771 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50905558 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13799920904192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120869108±2lµe4«)/ _wandbruntime‚11207  _runtime‚11207Ôœ•ç'¬)ŠN KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1987003161803182 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1973223285268480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1983424651522048 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1983424651522048 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436911893620736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22860513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28048818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120878174 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103470404 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13800592192000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224338587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103470406 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13800933538304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450860816248320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22860516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚450860929721856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50909597 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚13049 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224348580 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1997225226674688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1973223268229120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28049081 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13949036101120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50909331 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103470404 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348580 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13948941239808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13779876534702 _runtime‚11207.448443768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1983424634482688 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28049081 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13949036101120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50909331 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103470404 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348580 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13948941239808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13779876534702 _runtime‚11207.448443768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1983424634482688 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22860513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28048818 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120878174 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103470404 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13800592192000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224338587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103470406 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13800933538304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450860816248320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22860516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438²o]’14®)/ _wandbruntime‚11207  _runtime‚11207mï‘U ¯)† /loss/FRt1/detection_loss‚0.5439079506347018 1loss/SRqqt2/detection_loss‚0.5750171714232256 trainer/global_step‚13049 0loss/FRt1/assignment_loss‚1.5157849333297584 .loss/FBt1/detection_loss‚0.772229873375701 _step‚1075 /loss/FRt2/detection_loss‚1.4248983501814663 0loss/FBt2/detection_loss‚0.44728039800827685 %loss/total_loss‚7.632451188090936 epoch‚32 % _timestamp‚1.7843515487573807e+09 0loss/FRt2/assignment_loss‚0.5603641324990021 1loss/FBt1/assignment_loss‚0.10450878952404656 3loss/FBt2/assignment_loss‚0.0011396685030364109 0loss/SRqqt1/detection_loss‚0.805332586319816 _runtime‚11207.450679429 2loss/SRqqt1/assignment_loss‚0.4445016128218707 2loss/SRqqt2/assignment_loss‚0.4374857214700352$ã¯ß°)‹ 3loss/FBt2/assignment_loss‚0.0011396685030364109 0loss/SRqqt1/detection_loss‚0.805332586319816 _runtime‚11207.450679429 2loss/SRqqt1/assignment_loss‚0.4445016128218707 2loss/SRqqt2/assignment_loss‚0.4374857214700352 /loss/FRt1/detection_loss‚0.5439079506347018 1loss/SRqqt2/detection_loss‚0.5750171714232256 trainer/global_step‚13049 0loss/FRt1/assignment_loss‚1.5157849333297584 .loss/FBt1/detection_loss‚0.772229873375701 _step‚1075 /loss/FRt2/detection_loss‚1.4248983501814663 0loss/FBt2/detection_loss‚0.44728039800827685 %loss/total_loss‚7.632451188090936 epoch‚32 % _timestamp‚1.7843515487573807e+09 0loss/FRt2/assignment_loss‚0.5603641324990021 1loss/FBt1/assignment_loss‚0.10450878952404656³ÂJ‰=“±)‚(:ˆ É–ìÒ¬¼Á• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.683' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.5366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œã¡Nô²)‚(:é É–ìÒ”—ó– disk.rbd1.out‚ 38.3291015625 network.recv‚ 3.019837e+06 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚30018.50439453125 disk.dm-0.in‚18367.44140625 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚18367.44140625 disk.sda.out‚ 332.359375& disk.nvme0n1.out‚30018.50439453125( proc.memory.availableMB‚ 437.10546875) disk./.usagePercent‚39.28178990964941 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.1328125 cpu‚26.17658368639599 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0# disk.dm-0.out‚30018.50439453125) proc.memory.percent‚16.86558723449707 network.sent‚ 2.0814356e+07 disk.dm-1.out‚ 332.359375 disk.sda2.out‚0$ memory_percent‚97.33211994171143" proc.memory.rssMB‚ 2763.2578125$ disk./.usageGB‚585.3270568847656 disk.drbd2165.in‚0" disk.nvme0n1.in‚18367.44140625 disk.sda3.out‚ 332.359375cÇj“³)‚(:ˆ Ø–ìÒÎÎè” gpu.0.gpu‚42.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£›ß´)‚(:Ô Ø–ìÒ¿½¹– disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.out‚30021.75 disk.sda.out‚ 332.54296875 disk.sda2.in‚0 disk.nvme0n1.out‚30021.75 network.recv‚ 3.024306e+06$ disk./.usageGB‚585.3269271850586 disk.dm-1.out‚ 332.54296875 disk.drbd2165.in‚0 disk.dm-0.in‚18367.44140625 disk.sda3.out‚ 332.54296875) proc.memory.percent‚16.86558723449707 disk.dm-0.out‚30021.75 disk.sda.in‚ 0.1328125" proc.memory.rssMB‚ 2763.2578125 cpu‚26.17657258158452$ disk.nvme0c0n1.in‚18367.44140625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 network.sent‚ 2.0877523e+07 disk.rbd0.out‚0' proc.memory.availableMB‚ 436.8671875 disk.rbd0.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚18367.44140625 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 38.3291015625$ memory_percent‚97.33357429504395* disk./.usagePercent‚39.281781205393244 disk.dm-1.in‚ 0.1328125¹›Û~4µ)/ _wandbruntime‚11247  _runtime‚11247£®‹Ì¶)Æ trainer/global_step‚13099 % _timestamp‚1.7843515892528348e+09 _runtime‚11247.946716782 _step‚1076 ' lr-AdamW/pg1‚0.00012391709880527776 ' lr-AdamW/pg2‚0.00012391709880527776Ú-YÑ·)Ë % _timestamp‚1.7843515892528348e+09 _runtime‚11247.946716782 _step‚1076 ' lr-AdamW/pg1‚0.00012391709880527776 ' lr-AdamW/pg2‚0.00012391709880527776 trainer/global_step‚13099´9)¾Í4¸)/ _wandbruntime‚11247  _runtime‚11247z˜l(¹)P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103844374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1990639461974016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22946021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28150460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28150204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚1077 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22946018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103844374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13829141417886 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004489720001536 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1990639444934656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13850616406016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121323560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843515892539768e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452548901350912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13998998562304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1980402308190208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51096481 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121313583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452548792064512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51096222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1980402291150848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚11247.947796209 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2004489720001536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13829481967776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103844372 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438549884176384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1990639461974016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13850275066880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167934 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103844372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225157955 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121323560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13998907888128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13850616406016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13850275066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1994231432568734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¦Ä¸gÚ º)†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13998998562304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1980402308190208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51096481 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121313583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452548792064512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51096222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1980402291150848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚11247.947796209 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2004489720001536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13829481967776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103844372 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438549884176384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1990639461974016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13850275066880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167934 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103844372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225157955 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121323560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13998907888128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13850616406016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13850275066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1994231432568734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103844374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1990639461974016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22946021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28150460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28150204 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚1077 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22946018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103844374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13829141417886 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004489720001536 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1990639444934656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13850616406016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121323560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843515892539768e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452548901350912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0µr©&”4»)/ _wandbruntime‚11248  _runtime‚11248Ü_Ò'¼)‹N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13999943074816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103852004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438583312934912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51100005 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438583331547136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚13099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13851629040128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1990786685964288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452583256009728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843515898696544e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103852004 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121322635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1980548818862080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28152516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452583356900352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225174639 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121332626 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1990786685964288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121332626 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14000025353216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1980548801822720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1990786703003648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13851629040128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22947763 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2004637973658112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13830152471608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13851287693824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22947766 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225184632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103852006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121322635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0,S{'½)N trainer/global_step‚13099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13851629040128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1990786685964288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452583256009728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843515898696544e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103852004 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121322635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1980548818862080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28152516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452583356900352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225174639 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121332626 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1990786685964288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121332626 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14000025353216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1980548801822720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1990786703003648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13851629040128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22947763 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2004637973658112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13830152471608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13851287693824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22947766 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225184632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103852006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121322635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13999943074816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103852004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438583312934912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51100005 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438583331547136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576¶/˜À4¾)/ _wandbruntime‚11248  _runtime‚11248Yy9¿)‰ 2loss/SRqqt2/assignment_loss‚0.4477875303445915 1loss/SRqqt1/detection_loss‚0.7156337238094752 %loss/total_loss‚7.192065499276685 trainer/global_step‚13099 0loss/FRt1/assignment_loss‚1.5116179073761866 0loss/FRt2/assignment_loss‚0.6027665774701005 1loss/SRqqt2/detection_loss‚0.6890936156003121 /loss/FBt2/detection_loss‚0.3784682908233768 epoch‚32 _runtime‚11248.566608529 2loss/FBt2/assignment_loss‚0.005233766680111858 _step‚1079 3loss/FBt1/assignment_loss‚0.0014091305660661278 /loss/FRt1/detection_loss‚0.5545121266348833 /loss/FRt2/detection_loss‚1.3708603685427476 /loss/FBt1/detection_loss‚0.5020925680445018 % _timestamp‚1.7843515898733222e+09 3loss/SRqqt1/assignment_loss‚0.41258989338433155µž0À)Ž 3loss/FBt1/assignment_loss‚0.0014091305660661278 /loss/FRt1/detection_loss‚0.5545121266348833 /loss/FRt2/detection_loss‚1.3708603685427476 /loss/FBt1/detection_loss‚0.5020925680445018 % _timestamp‚1.7843515898733222e+09 3loss/SRqqt1/assignment_loss‚0.41258989338433155 2loss/SRqqt2/assignment_loss‚0.4477875303445915 1loss/SRqqt1/detection_loss‚0.7156337238094752 %loss/total_loss‚7.192065499276685 trainer/global_step‚13099 0loss/FRt1/assignment_loss‚1.5116179073761866 0loss/FRt2/assignment_loss‚0.6027665774701005 1loss/SRqqt2/detection_loss‚0.6890936156003121 /loss/FBt2/detection_loss‚0.3784682908233768 epoch‚32 _runtime‚11248.566608529 2loss/FBt2/assignment_loss‚0.005233766680111858 _step‚1079·mÀ4“Á)‚(:ˆ ç–ìÒÕ×»“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÊùÂ)‚(:î ç–ìÒ©¦û•) disk./.usagePercent‚39.28169928298229 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.33648300170898) proc.memory.percent‚16.86558723449707 cpu‚26.176362536760365$ disk./.usageGB‚585.3257064819336# disk.dm-0.out‚30027.09130859375 disk.sda.out‚ 332.83984375 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.0882439e+07 network.recv‚ 3.027742e+06 disk.dm-1.out‚ 332.83984375 disk.sda2.in‚0$ disk.nvme0c0n1.in‚18367.44140625( disk.nvme0c0n1.out‚30027.09130859375& disk.nvme0n1.out‚30027.09130859375 disk.sda1.out‚0& proc.memory.availableMB‚ 436.390625 disk.dm-0.in‚18367.44140625 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 332.83984375" disk.nvme0n1.in‚18367.44140625 disk.rbd1.out‚ 38.3916015625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2763.2578125T™Ñ·Ã)‚(:’ ö–ìÒÏ…ê“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.658' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.73159999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÔ`ºòÄ)‚(:ç ö–ìÒÈÒ¹–$ disk./.usageGB‚585.3256034851074 disk.drbd2165.out‚0) proc.memory.percent‚16.86558723449707 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 333.171875" disk.nvme0n1.in‚18367.44140625% disk.nvme0n1.out‚30031.1240234375 disk.rbd0.in‚0 disk.sda3.out‚ 333.171875 disk.rbd1.in‚ 2.01953125 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17 disk.sda1.in‚0' disk.nvme0c0n1.out‚30031.1240234375 disk.sda2.out‚0( proc.memory.availableMB‚ 437.06640625 cpu‚26.176014993884696) disk./.usagePercent‚39.28169237077886 disk.sda.in‚ 0.1328125$ memory_percent‚97.33235836029053 network.sent‚ 2.0945392e+07 disk.dm-1.out‚ 333.171875$ disk.nvme0c0n1.in‚18367.44140625 disk.dm-0.in‚18367.44140625" disk.dm-0.out‚30031.1240234375" proc.memory.rssMB‚ 2763.2578125 network.recv‚ 3.032047e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.rbd1.out‚ 38.3916015625$jÞÅžÅ)‚(:“ …—ìÒ¦ù“ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o÷#äÆ)‚(:Ù …—ìÒÔ—É–$ proc.memory.availableMB‚436.5625 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 30036.5859375 disk.dm-0.in‚18463.19921875 disk.sda.out‚ 333.578125" disk.nvme0n1.in‚18463.19921875 disk.rbd0.in‚0 network.sent‚ 2.0950531e+07 disk.drbd2165.in‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 30036.5859375 disk.sda3.out‚ 333.578125 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚18463.19921875 disk.sda.in‚ 0.1328125 cpu‚26.17586962452807$ memory_percent‚97.33543395996094) disk./.usagePercent‚39.28170568317065 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 30036.5859375 proc.cpu.threads‚17) proc.memory.percent‚16.86558723449707$ disk./.usageGB‚585.3258018493652 disk.rbd1.out‚ 38.4541015625 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 333.578125 disk.sda2.in‚0" proc.memory.rssMB‚ 2763.2578125 network.recv‚ 3.035309e+06 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0ŒæÍ4Ç)/ _wandbruntime‚11288  _runtime‚11288ÕÚ×IËÈ)Å ' lr-AdamW/pg1‚0.00012231163695822925 ' lr-AdamW/pg2‚0.00012231163695822925 trainer/global_step‚13149 $ _timestamp‚1.784351630262788e+09 _runtime‚11288.959475204 _step‚1080¤ÒÉÐÉ)Ê ' lr-AdamW/pg1‚0.00012231163695822925 ' lr-AdamW/pg2‚0.00012231163695822925 trainer/global_step‚13149 $ _timestamp‚1.784351630262788e+09 _runtime‚11288.959475204 _step‚1080¸ ¥ƒ~4Ê)/ _wandbruntime‚11288  _runtime‚11288!kõêþË)€P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104225972 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51286801 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2001607582649242 _runtime‚11288.960459352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1998001513455616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121778012 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104225972 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226003986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454271413210624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1998001496416256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900970568704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900970568704 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2011902825363456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225994007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 T¹ÑûÈFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13880098454684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 $ _timestamp‚1.784351630266762e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13901311907840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1998001496416256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454271301827072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28253794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440221322102784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1987727841783808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1998001513455616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14050072495616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13901311907840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2011902466984960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2001607940238492 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2011902825363456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104225974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104225972 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226003986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454271413210624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1998001496416256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900970568704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900970568704 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2011902825363456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225994007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13880098454684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 $ _timestamp‚1.784351630266762e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13901311907840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1998001496416256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454271301827072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28253794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440221322102784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1987727841783808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1998001513455616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14050072495616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13901311907840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2011902466984960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2001607940238492 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2011902825363456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104225974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BÆu4Í)/ _wandbruntime‚11289  _runtime‚11289“A('Î)‰N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902324541952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226010691 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454305862525952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1998148754485248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23035013 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28255841 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13880768958516 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13901983195648 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2001755104374836 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 # _timestamp‚1.78435163088317e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1998148754485248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚13149 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1998148737445888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121777087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23035016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1998148737445888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121787078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11289.576942015 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020684 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2012051079027200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51290584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28255571 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902324541952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104233604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440254769473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104233606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13881109508518 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020684 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14051004482560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1987874335416320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2012050720641536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104233606 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104233604 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440254750861312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14051093052416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0g²ÂH‡Ï)ŽN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104233604 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440254750861312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14051093052416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902324541952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226010691 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454305862525952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1998148754485248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23035013 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28255841 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13880768958516 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13901983195648 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2001755104374836 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 # _timestamp‚1.78435163088317e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1998148754485248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚13149 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1998148737445888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121777087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23035016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1998148737445888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121787078 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11289.576942015 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020684 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2012051079027200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51290584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28255571 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902324541952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104233604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚¬yà 440254769473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104233606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13881109508518 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020684 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14051004482560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1987874335416320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2012050720641536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104233606 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664ºEuä+4Ð)/ _wandbruntime‚11289  _runtime‚11289Nõ Ñ)† _runtime‚11289.579488863 3loss/FBt1/assignment_loss‚0.0008740464318779099 1loss/SRqqt1/detection_loss‚0.8626694320527108 0loss/SRqqt2/detection_loss‚0.584245186206559 &loss/total_loss‚7.6235111827027655 /loss/FRt1/detection_loss‚0.5746728818563983 epoch‚32 trainer/global_step‚13149 /loss/FRt1/assignment_loss‚1.499753335501888 /loss/FRt2/assignment_loss‚0.623794325987498 2loss/SRqqt1/assignment_loss‚0.5514313366995633 3loss/SRqqt2/assignment_loss‚0.49656330951089983 _step‚1083 4loss/FBt2/assignment_loss‚2.8702935545030927e-05 .loss/FRt2/detection_loss‚1.433307383880278 /loss/FBt1/detection_loss‚0.5406405723808955 /loss/FBt2/detection_loss‚0.4555306692586524 # _timestamp‚1.78435163088617e+09 nséÒ)‹ _runtime‚11289.579488863 3loss/FBt1/assignment_loss‚0.0008740464318779099 1loss/SRqqt1/detection_loss‚0.8626694320527108 0loss/SRqqt2/detection_loss‚0.584245186206559 &loss/total_loss‚7.6235111827027655 /loss/FRt1/detection_loss‚0.5746728818563983 epoch‚32 trainer/global_step‚13149 /loss/FRt1/assignment_loss‚1.499753335501888 /loss/FRt2/assignment_loss‚0.623794325987498 2loss/SRqqt1/assignment_loss‚0.5514313366995633 3loss/SRqqt2/assignment_loss‚0.49656330951089983 _step‚1083 4loss/FBt2/assignment_loss‚2.8702935545030927e-05 .loss/FRt2/detection_loss‚1.433307383880278 /loss/FBt1/detection_loss‚0.5406405723808955 /loss/FBt2/detection_loss‚0.4555306692586524 # _timestamp‚1.78435163088617e+09»¹ñg¡žÓ)‚(:“ ”—ìÒâµ” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚209.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.964600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Jr{úÔ)‚(:ï ”—ìÒ¤µÎ–" proc.memory.rssMB‚ 2763.2578125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚ 3.038222e+06 disk.drbd2165.in‚0) proc.memory.percent‚16.86558723449707 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.rbd1.out‚ 38.4541015625 disk.rbd0.in‚0 network.sent‚ 2.0955415e+07) disk./.usagePercent‚39.28159483190832" disk.nvme0n1.in‚18463.19921875$ disk.nvme0c0n1.in‚18463.19921875& disk.nvme0n1.out‚30039.44580078125 disk.sda3.out‚ 334.19140625( proc.memory.availableMB‚ 436.30078125( disk.nvme0c0n1.out‚30039.44580078125 disk.sda.in‚ 0.1328125 disk.sda.out‚ 334.19140625 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0 cpu‚26.17559204431522$ memory_percent‚97.33703136444092$ disk./.usageGB‚585.3241500854492 disk.dm-1.out‚ 334.19140625 disk.dm-0.in‚18463.19921875# disk.dm-0.out‚30039.44580078125 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0L(™¤Õ)‚(:’ £—ìÒÑÀõ“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐaéüÖ)‚(:ñ £—ìÒ”¼ù–( disk.nvme0c0n1.out‚30046.66748046875 disk.dm-0.in‚18463.19921875 disk.sda3.in‚ 0.1328125# disk.dm-0.out‚30046.66748046875$ disk.nvme0c0n1.in‚18463.19921875 disk.sda.out‚ 334.51953125" disk.nvme0n1.in‚18463.19921875 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0* disk./.usagePercent‚39.281596367953526 disk.sda3.out‚ 334.51953125 disk.drbd2165.out‚0 network.recv‚ 3.043124e+06 disk.rbd0.in‚0& disk.nvme0n1.out‚30046.66748046875( proc.memory.availableMB‚ 436.48828125" proc.memory.rssMB‚ 2763.2578125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.1018415e+07$ disk./.usageGB‚585.3241729736328 disk.dm-1.out‚ 334.51953125$ memory_percent‚97.33588695526123) proc.memory.percent‚16.86558723449707 cpu‚26.175402323375987 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 38.5791015625 disk.dm-1.in‚ 0.1328125¿T•Ù‘×)‚(:† ²—ìÒò×Ö” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„@<žôØ)‚(:é ²—ìÒÚÊÆ• disk.rbd1.out‚ 38.5791015625 disk.sda3.in‚ 0.1328125$ memory_percent‚97.33719825744629 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚30050.990234375 disk.sda.out‚ 334.88671875' proc.memory.availableMB‚ 436.2734375) disk./.usagePercent‚39.28159739198367$ disk./.usageGB‚585.3241882324219 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.1023418e+07$ disk.nvme0c0n1.in‚18463.19921875 cpu‚26.175133220173016 network.recv‚ 3.046007e+06 disk.sda1.out‚0 disk.dm-1.out‚ 334.88671875 disk.drbd2165.in‚0 disk.dm-0.in‚18463.19921875 disk.sda2.in‚0$ disk.nvme0n1.out‚30050.990234375 disk.sda3.out‚ 334.88671875 disk.dm-1.in‚ 0.1328125! disk.dm-0.out‚30050.990234375" proc.memory.rssMB‚ 2763.2578125) proc.memory.percent‚16.86558723449707 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚18463.19921875â‚Õñ4Ù)/ _wandbruntime‚11329  _runtime‚11329`²™ÊÚ)Ä trainer/global_step‚13199 % _timestamp‚1.7843516710626934e+09 _runtime‚11329.760421713 _step‚1084 & lr-AdamW/pg1‚0.0001207127409497261 & lr-AdamW/pg2‚0.0001207127409497261*ÃëÏÛ)É trainer/global_step‚13199 % _timestamp‚1.7843516710626934e+09 _runtime‚11329.760421713 _step‚1084 & lr-AdamW/pg1‚0.0001207127409497261 & lr-AdamW/pg2‚0.0001207127409497261¼¿³ò¥4Ü)/ _wandbruntime‚11329  _runtime‚11329*¢H(Ý)P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019315213968384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51477196 _runtime‚11329.763596716 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1085 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019315213968384 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2005363564937216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14101138349056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2019315572346880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122232464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28356678 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13952007409664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13951666070528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2019315572346880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23120518 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14101056063488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23120521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28356950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455993816092672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚13199 % _timestamp‚1.7843516710698938e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019315213968384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51477196 _runtime‚11329.763596716 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1085 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019315213968384 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2005363564937216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14101138349056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2019315572346880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122232464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28356678 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13952007409664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13951666070528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2019315572346880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23120518 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14101056063488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23120521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28356950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455993816092672½†Æí4ß)/ _wandbruntime‚11330  _runtime‚11330|ö¦à)ŠN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23122263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122231539 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846743 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931385445424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13953020043776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019463826010624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13952678697472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2019463826010624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456028374800896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51480970 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14102167400960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11330.375940843 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28358966 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13953020043776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1086 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843516716821873e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122241530 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122231539 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122241530 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51481232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2009131254455344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441926207399936 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2009131612044706 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104615206 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2005510788927488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1995199886049280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13952678697472¢D$œ'á)N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019463826010624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13952678697472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2019463826010624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456028374800896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51480970 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14102167400960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11330.375940843 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28358966 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13953020043776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1086 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843516716821873e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122241530 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122231539 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122241530 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51481232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2009131254455344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441926207399936 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2009131612044706 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104615206 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2005510788927488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1995199886049280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13952678697472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23122263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122231539 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846743 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931385445424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13953020043776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 &particle/accuracy‚0.83769690990448 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3350536823272705 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2894652783870697 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026769092306494713 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2323438823223114 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 'particle/f_score‚0.6154248714447021 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6986326575279236 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938676953315735 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6054471135139465 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7182579636573792 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9829427003860474 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 *loss/val_total_loss‚7.6347517904516655 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958154559135437 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3604155480861664 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 +jet/accuracy_2_of_2‚0.13576433062553406 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6124632954597473 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7654035687446594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7934048771858215 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7360010147094727 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8399092555046082 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714311003684998 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_3‚0.9211242198944092 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8079431653022766 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6606534123420715 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24800491333007812 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3267468512058258 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2782502770423889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026988878846168518 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2090333104133606 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.712278425693512 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5329166650772095 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8828411102294922 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994443416595459 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030234303325414658 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7922272682189941 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22264592349529266 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6286061406135559 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2286403477191925 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6842147707939148 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6508336067199707 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6219327449798584 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2277829498052597 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998930096626282 trainer/global_step‚13199 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21075980365276337 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08281020820140839 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8828411102294922 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803458213806152 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24183781445026398 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2776288092136383 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18341803550720215 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 /particle/accuracy_4_of_4‚0.9353388547897339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 epoch‚32 % _timestamp‚1.7843516905454223e+09 _runtime‚11349.239654987 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2393745481967926 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7780669331550598 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9426186084747314 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038156550377607346 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.04746416211128235 +particle/specificity‚0.9014045000076294 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949389696121216 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24592463672161102 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966593384742737 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029876140877604485 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23825354874134064 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7109938859939575 +particle/sensitivity‚0.6047521829605103 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6624093651771545 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7291046977043152 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03542894124984741 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01947816275060177 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7733938097953796 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3674117922782898 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695725321769714 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09502075612545013 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9893653392791748 *jet/accuracy_2_of_3‚0.5251809358596802 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4975613057613373 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04302499070763588 particle/accuracy_1_of_4‚1 _step‚1088 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5637020468711853 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8413076996803284 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10869305580854416 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6828498244285583 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23671944439411163 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12634651362895966 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0‰ïîo0FRt*SRqqt0FBt/SRqqt_purity‚0.6397546529769897 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5171359181404114 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 *jet/accuracy_1_of_2‚0.7725169658660889 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7587986588478088 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5323801636695862 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6615414023399353 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6358148455619812 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6227049827575684 .particle/accuracy_3_of_4‚0.997619092464447 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01947816275060177 particle/accuracy_5_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.804365873336792 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8505479693412781 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10165601968765259 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3413013517856598 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 )jet/accuracy_2_of_4‚0.779374361038208 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955142736434937 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2776288092136383 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7135268449783325 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6206262707710266 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.143593892455101 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23926366865634918 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28399473428726196 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22725112736225128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26926788687705994 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4939475357532501 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30081698298454285 7validation_average_jet_accuracy‚0.37341198325157166 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1257622241973877 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27998414635658264 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7833848595619202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049314506351947784 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7131452560424805 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6383740901947021*W«ŸÁ=é)»{ :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30819782614707947 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8619046807289124 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6346432566642761 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7626559734344482 *jet/accuracy_1_of_1‚0.3381423354148865 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7097018361091614 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8565003275871277 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 &particle/accuracy‚0.83769690990448 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3350536823272705 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2894652783870697 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026769092306494713 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2323438823223114 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 'particle/f_score‚0.6154248714447021 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6986326575279236 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938676953315735 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6054471135139465 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7182579636573792 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9829427003860474 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 *loss/val_total_loss‚7.6347517904516655 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958154559135437 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3604155480861664 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 +jet/accuracy_2_of_2‚0.13576433062553406 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6124632954597473 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7654035687446594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7934048771858215 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7360010147094727 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8399092555046082 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714311003684998 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_3‚0.9211242198944092 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8079431653022766 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6606534123420715 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24800491333007812 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3267468512058258 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2782502770423889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026988878846168518 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2090333104133606 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.712278425693512 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5329166650772095 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8828411102294922 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994443416595459 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030234303325414658 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7922272682189941 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22264592349529266 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6286061406135559 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2286403477191925 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6842147707939148 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6508336067199707 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6219327449798584 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2277829498052597 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998930096626282 trainer/global_step‚13199 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21075980365276337 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08281020820140839 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8828411102294922 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803458213806152 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24183781445026398 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2776288092136383 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18341803550720215 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11749999225139618 /particle/accuracy_4_of_4‚0.9353388547897339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 epoch‚32 % _timestamp‚1.7843516905454223e+09 _runtime‚11349.239654987 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2393745481967926 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7780669331550598 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9426186084747314 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038156550377607346 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.04746416211128235 +particle/specificity‚0.9014045000076294 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949389696121216 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24592463672161102 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966593384742737 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029876140877604485 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23825354874134064 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7109938859939575 +particle/sensitivity‚0.6047521829605103 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6624093651771545 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7291046977043152 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03542894124984741 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01947816275060177 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7733938097953796 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3674117922782898 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695725321769714 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09502075612545013 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9893653392791748 *jet/accuracy_2_of_3‚0.5251809358596802 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4975613057613373 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04302499070763588 particle/accuracy_1_of_4‚1 _step‚1088 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5637020468711853 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8413076996803284 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10869305580854416 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6828498244285583 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23671944439411163 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12634651362895966 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6397546529769897 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5171359181404114 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 *jet/accuracy_1_of_2‚0.7725169658660889 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7587986588478088 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5323801636695862 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6615414023399353 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6358148455619812 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6227049827575684 .particle/accuracy_3_of_4‚0.997619092464447 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01947816275060177 particle/accuracy_5_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.804365873336792 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8505479693412781 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10165601968765259 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3413013517856598 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 )jet/accuracy_2_of_4‚0.779374361038208 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955142736434937 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2776288092136383 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7135268449783325 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6206262707710266 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.143593892455101 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23926366865634918 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28399473428726196 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22725112736225128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26926788687705994 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4939475357532501 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30081698298454285 7validation_average_jet_accuracy‚0.37341198325157166 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1257622241973877 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27998414635658264 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7833848595619202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049314506351947784 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7131452560424805 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6383740901947021 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976933002471924 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6219327449798584 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.72853684425354 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2883489429950714 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.046347443014383316 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313À„iÚÊ“ê)‚(:ˆ ЗìÒ茥” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤6dÂÖë)‚(:Ë ЗìÒÀª¢– disk.rbd1.out‚ 38.6416015625$ memory_percent‚97.44257926940918* proc.memory.percent‚16.970062255859375 disk.sda3.out‚ 335.421875( disk.nvme0c0n1.out‚30062.53369140625 disk.sda2.out‚0' proc.memory.availableMB‚ 419.0078125 network.recv‚ 3.053973e+06 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 335.421875 disk.nvme0c0n1.in‚18548$ disk./.usageGB‚585.3280754089355 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0# disk.dm-0.out‚30062.53369140625 disk.sda.out‚ 335.421875 disk.sda2.in‚0 network.sent‚ 2.1091706e+07 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚18548& disk.nvme0n1.out‚30062.53369140625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.dm-0.in‚18548) disk./.usagePercent‚39.28185826366105 disk.drbd2165.in‚0 proc.memory.rssMB‚2780.375 cpu‚26.178818879248087;í{y‘ì)‚(:† ß—ìÒýñÑ“ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚248.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞWVÚÛí)‚(:Ð ß—ìÒ…²å• disk.sda2.in‚0% disk.nvme0n1.out‚30068.6357421875$ memory_percent‚97.44255542755127 cpu‚26.178629726979587 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 proc.memory.rssMB‚2780.375 disk.nvme0c0n1.in‚18548 disk.dm-0.in‚18548" disk.dm-0.out‚30068.6357421875 disk.nvme0n1.in‚18548 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 336.00390625 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 38.7041015625( proc.memory.availableMB‚ 419.01171875 disk.sda1.out‚0 network.sent‚ 2.1149012e+07 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚30068.6357421875 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 336.00390625 disk.rbd1.in‚ 2.01953125* disk./.usagePercent‚39.281866711909686$ disk./.usageGB‚585.3282012939453* proc.memory.percent‚16.970062255859375 network.recv‚ 3.058379e+06 disk.dm-1.out‚ 336.00390625 disk.rbd0.in‚0 disk.drbd2165.out‚0î[Ì’î)‚(:‡ î—ìÒôû” gpu.0.gpu‚24.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l.‹Óï)‚(:È î—ìÒ•Œ– disk.drbd2165.out‚0 proc.memory.rssMB‚2780.375* proc.memory.percent‚16.970062255859375& disk.nvme0n1.out‚30072.43896484375 disk.rbd1.out‚ 38.7041015625 disk.dm-1.out‚ 336.28125( disk.nvme0c0n1.out‚30072.43896484375$ memory_percent‚97.44269847869873 disk.sda2.out‚0 disk.sda3.out‚ 336.28125 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.28185621560078 disk.sda3.in‚ 0.1328125 cpu‚26.17834970973585 proc.cpu.threads‚17 network.sent‚ 2.1153968e+07 network.recv‚ 3.061432e+06( proc.memory.availableMB‚ 418.98828125# disk.dm-0.out‚30072.43896484375 disk.sda.out‚ 336.28125 disk.nvme0c0n1.in‚18548 disk.sda2.in‚0 disk.nvme0n1.in‚18548 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125$ disk./.usageGB‚585.3280448913574 disk.dm-0.in‚18548 disk.sda.in‚ 0.1328125 disk.sda1.out‚0 disk.drbd2165.in‚0Pö½š4ð)/ _wandbruntime‚11389  _runtime‚11389I’ÝžÌñ)Æ ' lr-AdamW/pg1‚0.00011912051347387099 ' lr-AdamW/pg2‚0.00011912051347387099 trainer/global_step‚13249 % _timestamp‚1.7843517307777214e+09 _runtime‚11389.471339728 _step‚1089Ðüå6Ñò)Ë trainer/global_step‚13249 % _timestamp‚1.7843517307777214e+09 _runtime‚11389.471339728 _step‚1089 ' lr-AdamW/pg1‚0.00011912051347387099 ' lr-AdamW/pg2‚0.00011912051347387099Á¤ä4ó)/ _wandbruntime‚11389  _runtime‚11389åÖ O6 ô)€P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14020436116992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14170481084928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23210288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11389.472428363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23210291 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14020094777856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14170400896512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2028675260807680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2014655183069184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51700988 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457750510951936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚13249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122757238 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122757238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457750609752576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13998710603839 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2028675619186176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105109840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443580128667648 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2018286623338817 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443580110055424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14020094777856 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11389.472428363 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23210291 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14020094777856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14170400896512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2028675260807680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2014655183069184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51700988 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457750510951936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438Â4²o†4ö)/ _wandbruntime‚11390  _runtime‚11390…åê'÷)‹N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14171421179904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105117470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23212036 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443613538813952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105117472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28492992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122766304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014802407059456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105117472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2028823514464256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457784959993856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1091 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14000062207563 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23212033 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2028823872849920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51704754 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2004434065817600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2014802407059456 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2018433787475161 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2004434082856960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2018434145064523 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2014802424098816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122756313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105117470 % _timestamp‚1.7843517313970046e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2014802424098816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122766304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13999721657561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883776 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873783 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28492721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883776 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873783 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28492721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105117470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23212036 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443613538813952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105117472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28492992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122766304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014802407059456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105117472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2028823514464256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457784959993856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1091 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14000062207563 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23212033 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2028823872849920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51704754 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2004434065817600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2014802407059456 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2018433787475161 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2004434082856960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2018434145064523 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2014802424098816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122756313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105117470 % _timestamp‚1.7843517313970046e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2014802424098816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122766304ÃÓáz4ù)/ _wandbruntime‚11390  _runtime‚11390󓞺eú)ˆ 3loss/FBt1/assignment_loss‚5.803751059098224e-05 2loss/FBt2/assignment_loss‚0.009266267006013653 1loss/SRqqt1/detection_loss‚0.7426819236745359 1loss/SRqqt2/detection_loss‚0.5733425035369953 /loss/FBt1/detection_loss‚0.8958828265792417 0loss/FRt1/assignment_loss‚1.4476326031984317 0loss/FRt2/assignment_loss‚0.69027736043pN©35555 epoch‚33 trainer/global_step‚13249 _step‚1092 /loss/FRt2/detection_loss‚1.4729547071890592 % _timestamp‚1.7843517314001124e+09 _runtime‚11390.093403797 /loss/FRt1/detection_loss‚0.4973307782449971 %loss/total_loss‚7.657190184710742 .loss/FBt2/detection_loss‚0.401558758530225 2loss/SRqqt1/assignment_loss‚0.4598506466785565 3loss/SRqqt2/assignment_loss‚0.46635377212853946®¤$žû) /loss/FRt2/detection_loss‚1.4729547071890592 % _timestamp‚1.7843517314001124e+09 _runtime‚11390.093403797 /loss/FRt1/detection_loss‚0.4973307782449971 %loss/total_loss‚7.657190184710742 .loss/FBt2/detection_loss‚0.401558758530225 2loss/SRqqt1/assignment_loss‚0.4598506466785565 3loss/SRqqt2/assignment_loss‚0.46635377212853946 3loss/FBt1/assignment_loss‚5.803751059098224e-05 2loss/FBt2/assignment_loss‚0.009266267006013653 1loss/SRqqt1/detection_loss‚0.7426819236745359 1loss/SRqqt2/detection_loss‚0.5733425035369953 /loss/FBt1/detection_loss‚0.8958828265792417 0loss/FRt1/assignment_loss‚1.4476326031984317 0loss/FRt2/assignment_loss‚0.6902773604335555 epoch‚33 trainer/global_step‚13249 _step‚1092Ä3µïs‘ü)‚(:† ý—ìÒËšá” gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àð‡‘óý)‚(:è ý—ìÒ²€Ç– disk.rbd1.out‚ 38.7666015625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚30080.7021484375 disk.sda.in‚ 0.1328125) disk./.usagePercent‚39.28186210377407 disk.dm-0.in‚18599.47265625$ disk./.usageGB‚585.3281326293945 disk.sda.out‚ 336.70703125 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚18599.47265625" disk.dm-0.out‚30080.7021484375 proc.memory.rssMB‚2780.375 cpu‚26.17809155405811 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚16.970062255859375' proc.memory.availableMB‚ 418.9296875 network.recv‚ 3.064787e+06" disk.nvme0n1.in‚18599.47265625% disk.nvme0n1.out‚30080.7021484375 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 network.sent‚ 2.1158968e+07 disk.sda3.out‚ 336.70703125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 336.70703125 disk.drbd2165.in‚0$ memory_percent‚97.44305610656738 disk.sda3.in‚ 0.1328125ØLþ)‚(:’ Œ˜ìÒ‹¦š” gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëoïÿ)‚(:ä Œ˜ìÒ·Ý—* proc.memory.percent‚16.970062255859375 cpu‚26.177833582780696 network.sent‚ 2.1222202e+07 disk.sda.out‚ 337.24609375 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 network.recv‚ 3.069039e+06$ disk.nvme0n1.out‚30085.365234375 disk.sda3.out‚ 337.24609375 disk.rbd0.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚30085.365234375! disk.dm-0.out‚30085.365234375 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0% proc.memory.availableMB‚ 418.90625$ disk./.usageGB‚585.3280334472656 disk.dm-1.out‚ 337.24609375 proc.memory.rssMB‚2780.375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚18599.47265625 disk.dm-0.in‚18599.47265625$ memory_percent‚97.44319915771484) disk./.usagePercent‚39.28185544757818 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚18599.47265625 disk.rbd1.out‚ 38.7666015625ÿ7ž€*‚(:“ ›˜ìÒÜâÉ” gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚254.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.962799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÀ“Éï*‚(:ä ›˜ìÒ— disk.sda.out‚ 337.58984375 disk.sda2.in‚0& disk.nvme0c0n1.out‚30092.572265625! disk.dm-0.out‚30092.572265625 disk.sda2.out‚0" disk.nvme0n1.in‚18599.47265625$ disk.nvme0n1.out‚30092.572265625 disk.sda3.out‚ 337.58984375 disk.rbd1.out‚ 38.8291015625 proc.memory.rssMB‚2780.375 disk.drbd2165.out‚0 network.recv‚ 3.07252e+06 disk.rbd0.in‚0* proc.memory.percent‚16.970062255859375' proc.memory.availableMB‚ 418.8203125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚18599.47265625 disk.dm-1.out‚ 337.58984375 proc.cpu.threads‚17 disk.dm-0.in‚18599.47265625$ memory_percent‚97.44372367858887 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 cpu‚26.177708635550086 disk.rbd1.in‚ 2.01953125 network.sent‚ 2.1227116e+07) disk./.usagePercent‚39.28185391153297# disk./.usageGB‚585.328010559082 disk.rbd0.out‚0 disk.sda1.in‚0hÐ’84‚*/ _wandbruntime‚11430  _runtime‚11430·~ŤËƒ*Å $ _timestamp‚1.784351771788815e+09 _runtime‚11430.482338776 _step‚1093 ' lr-AdamW/pg1‚0.00011753505679645935 ' lr-AdamW/pg2‚0.00011753505679645935 trainer/global_step‚13299Dz{Є*Ê ' lr-AdamW/pg1‚0.00011753505679645935 ' lr-AdamW/pg2‚0.00011753505679645935 trainer/global_step‚13299 $ _timestamp‚1.784351771788815e+09 _runtime‚11430.482338776 _step‚1093ÅR¹#4…*/ _wandbruntime‚11430  _runtime‚11430K]^³(†*P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123211690 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459473377421312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51891864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2036088007791104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28594596 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445251547981824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14049667640637 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28594326 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚13299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14070790279680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2011613088739328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1094 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2036088366169600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2022017217511424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105491440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445251566594048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123201713 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2025662415830075 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123201713 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51891864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2036088007791104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28594596 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445251547981824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14049667640637 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28594326 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚13299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14070790279680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2011613088739328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1094 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2036088366169600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2022017217511424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105491440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445251566594048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123201713 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2025662415830075 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123201713 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51895616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _t˜øÀ‡$imestamp‚1.7843517724025714e+09 _step‚1095 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2036236619833344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14072144252928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459507831045632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14222835693056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14222751317504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14071802906624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719828 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2022164458541056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123220756 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123210765 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2025810295145031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚11431.096514298 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14050338144469 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105499070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚11691Xq\'Š*N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2022164475580416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14050678694471 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843517724025714e+09 _step‚1095 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2036236619833344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14072144252928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459507831045632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14222835693056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14222751317504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14071802906624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719828 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2022164458541056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123220756 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123210765 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2025810295145031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚11431.096514298 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14050338144469 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105499070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51895616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52082234 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14121485781504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14100284127545 % _timestamp‚1.7843518126225104e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2043500754774528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2029379286032384 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123666142 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14272954725376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2043500754774528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123656165 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2033038565910583 trainer/global_step‚13349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529203 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105873040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229539182 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105873040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28697446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2043501113153024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14121485781504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‹àQï —*†P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123666142 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14272853565440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461195959245824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461195839473664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2018938639372288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23384788 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28697716 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23384791 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123656165 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14121827120640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105873038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2029379268993024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105873038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2029379268993024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14121827120640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20189386/“22332928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446922985908224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚11471.31630019 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2043501113153024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123666142 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14272954725376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2043500754774528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123656165 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2033038565910583 trainer/global_step‚13349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529203 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105873040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229539182 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105873040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28697446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2043501113153024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14121485781504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52082234 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14121485781504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14100284127545 % _timestamp‚1.7843518126225104e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2043500754774528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2029379286032384 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123675208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14101295181379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446956433278976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14122839754752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2043649008431104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14122839754752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52086034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2033186087636177 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123675208 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105880670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2019085150044160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2019085133004800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2029526510022656 % _timestamp‚1.7843518132393956e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14273967548928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14273872687616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2043649366816768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23386533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446956414666752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229545887 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123665217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23386536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229555880’D ¡'š*N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28699501 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123675208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14101295181379 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446956433278976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14122839754752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2043649008431104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14122839754752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52086034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2033186087636177 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123675208 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105880670 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2019085150044160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2019085133004800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2029526510022656 % _timestamp‚1.7843518132393956e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14273967548928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14273872687616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2043649366816768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23386533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446956414666752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229545887 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123665217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23386536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229555880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461230287354368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14122498408448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2043649366816768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2029526527062016 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2029526510022656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚11471.933237644 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123665217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229545887 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105880670 trainer/global_step‚13349 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2033186445225539ËÑD Ð4›*/ _wandbruntime‚11471  _runtime‚11471vøý œ*… 0loss/FRt1/assignment_loss‚1.4824829829272939 /loss/FRt2/assignment_loss‚0.675154644818533 /loss/FBt2/detection_loss‚0.4726502884737716 % _timestamp‚1.7843518132421608e+09 3loss/FBt1/assignment_loss‚0.0020344746148961783 /loss/FRt1/detection_loss‚0.5460402696154047 1loss/SRqqt2/detection_loss‚0.6297915963696409 0loss/FBt1/detection_loss‚0.39272730948493606 epoch‚33 _runtime‚11471.935438289 _step‚1100 1loss/SRqqt1/assignment_loss‚0.552291473353169 2loss/SRqqt2/assignment_loss‚0.4084943015297138 2loss/FBt2/assignment_loss‚0.003496001987623216 /loss/FRt2/detection_loss‚1.4423847506959213 #loss/total_loss‚7.4093521782297 trainer/global_step‚13349 1loss/SRqqt1/detection_loss‚0.8018040843587958ú#Ü*Š 2loss/FBt2/assignment_loss‚0.003496001987623216 /loss/FRt2/detection_loss‚1.4423847506959213 #loss/total_loss‚7.4093521782297 trainer/global_step‚13349 1loss/SRqqt1/detection_loss‚0.8018040843587958 0loss/FRt1/assignment_loss‚1.4824829829272939 /loss/FRt2/assignment_loss‚0.675154644818533 /loss/FBt2/detection_loss‚0.4726502884737716 % _timestamp‚1.7843518132421608e+09 3loss/FBt1/assignment_loss‚0.0020344746148961783 /loss/FRt1/detection_loss‚0.5460402696154047 1loss/SRqqt2/detection_loss‚0.6297915963696409 0loss/FBt1/detection_loss‚0.39272730948493606 epoch‚33 _runtime‚11471.935438289 _step‚1100 1loss/SRqqt1/assignment_loss‚0.552291473353169 2loss/SRqqt2/assignment_loss‚0.4084943015297138ÌóQ-ž*‚(:’ ȘìÒ ø” gpu.0.gpu‚92.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯[òëŸ*‚(:à ȘìÒžŽ— disk.rbd1.out‚ 38.8916015625$ memory_percent‚97.44372367858887 disk.drbd2165.out‚0# disk.dm-0.out‚30106.28369140625 disk.sda.in‚ 0.1328125$ disk./.usageGB‚585.3279609680176! disk.nvme0n1.in‚ 18659.8046875 disk.sda3.in‚ 0.1328125 disk.drbd2165.in‚0 cpu‚26.177275454695224 disk.sda.out‚ 338.53125' proc.memory.availableMB‚ 418.8203125* proc.memory.percent‚16.970062255859375& disk.nvme0n1.out‚30106.28369140625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30106.28369140625 disk.dm-0.in‚ 18659.8046875 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 338.53125 proc.cpu.threads‚17 network.sent‚ 2.1313336e+07 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 338.53125# disk.nvme0c0n1.in‚ 18659.8046875 proc.memory.rssMB‚2780.375 network.recv‚ 3.084099e+06) disk./.usagePercent‚39.28185058343502ók™“ *‚(:ˆ טìÒáž°“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷œÂÉí¡*‚(:â טìÒ̱Η% disk.nvme0n1.out‚30110.0693359375 disk.rbd1.out‚ 39.0166015625* disk./.usagePercent‚39.281861079743926 disk.sda3.out‚ 339.109375 proc.cpu.threads‚17 disk.dm-1.out‚ 339.109375 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 proc.memory.rssMB‚2780.375* proc.memory.percent‚16.970062255859375 cpu‚26.177064053834084 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 18659.8046875 disk.dm-0.in‚ 18659.8046875 disk.sda2.in‚0 network.recv‚ 3.088519e+06 disk.sda1.out‚0! disk.nvme0n1.in‚ 18659.8046875 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.43106365203857( proc.memory.availableMB‚ 420.89453125 network.sent‚ 2.1363382e+07$ disk./.usageGB‚585.3281173706055 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125' disk.nvme0c0n1.out‚30110.0693359375" disk.dm-0.out‚30110.0693359375 disk.sda.out‚ 339.109375 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.in‚0dKIJž¢*‚(:“ æ˜ìÒ¯¼¿“ gpu.0.gpu‚98.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S†9dç£*‚(:Ü æ˜ìÒüÿÿ• disk.drbd2165.out‚0 disk.dm-0.out‚30113.58203125 disk.sda2.out‚0 proc.memory.rssMB‚2780.375 cpu‚26.176949739590803) disk./.usagePercent‚39.28186363981927 disk.sda2.in‚0 disk.sda3.out‚ 339.3203125$ memory_percent‚97.43103981018066$ disk./.usageGB‚585.3281555175781 disk.dm-1.out‚ 339.3203125 disk.sda.out‚ 339.3203125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 39.0166015625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 18659.8046875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 18659.8046875% disk.nvme0c0n1.out‚30113.58203125 disk.rbd1.in‚ 2.01953125 network.recv‚ 3.09194e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18659.8046875# disk.nvme0n1.out‚30113.58203125 proc.cpu.threads‚17 network.sent‚ 2.1368411e+07 disk.sda.in‚ 0.1328125' proc.memory.availableMB‚ 420.8984375* proc.memory.percent‚16.970062255859375 disk.drbd2165.in‚0‡»rN4¤*/ _wandbruntime‚11512  _runtime‚11512ÈÖÑ¥*Å ' lr-AdaÀ“à¼amW/pg1‚0.00011438486271922936 ' lr-AdamW/pg2‚0.00011438486271922936 trainer/global_step‚13399 $ _timestamp‚1.784351853682803e+09 _runtime‚11512.376405858 _step‚1101àm[<Ц*Ê _step‚1101 ' lr-AdamW/pg1‚0.00011438486271922936 ' lr-AdamW/pg2‚0.00011438486271922936 trainer/global_step‚13399 $ _timestamp‚1.784351853682803e+09 _runtime‚11512.376405858Ís©@Ë4§*/ _wandbruntime‚11512  _runtime‚11512ƒvl(¨*P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448594442446848 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2040415073580341 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462918501827072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚13399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230365255 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036741320474624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106254638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23472038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843518536840234e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14172522622464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2026264155926528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124120594 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14172522622464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23472041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462918384152064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2036741337513984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2050913860136448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2026264172965888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2050913860136448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14172181283328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448594423834624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106254640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2050913501757952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124110617 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544ʪÄf (©*†P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2036741337513984 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2050913860136448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2026264172965888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2050913860136448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14172181283328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448594423834624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106254640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2050913501757952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124110617 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448594442446848 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2040415073580341 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462918501827072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚13399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230365255 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036741320474624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106254638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23472038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843518536840234e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14172522622464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2026264155926528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124120594 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14172522622464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23472041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462918384152064 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14324990142464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448627852593152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚11512.99247427 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14173535256576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124129660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462952947820544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52276108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚13399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23473783 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106262270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23473786 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391932 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448627871205376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124119669 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106262272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2036888578543616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124129660 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14173535256576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106262270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2051062113800192 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2036888561504256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462952842735616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051061755414528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2040562237716685 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2026410683637760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381939 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2036888561504256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391932 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14325076615168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14173193910272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ˜ºã³¬*N SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391932 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448627871205376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124119669 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106262272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2036888578543616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124129660 C=DeviceStatsž¼¹` Monitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14173535256576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106262270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2051062113800192 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2036888561504256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462952842735616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051061755414528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2040562237716685 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2026410683637760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381939 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2036888561504256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391932 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14325076615168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14173193910272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2036888578543616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14173193910272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28802325 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 # _timestamp‚1.78435185429869e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14324990142464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448627852593152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚11512.99247427 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14173535256576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124129660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462952947820544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52276108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚13399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23473783 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106262270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23473786ÏÉËc4­*/ _wandbruntime‚11512  _runtime‚11512ÊKÐm®*Š 0loss/FRt2/assignment_loss‚0.5591389108146703 1loss/SRqqt2/detection_loss‚0.5736395215673665 /loss/FBt2/detection_loss‚0.2805849972452658 %loss/total_loss‚6.898898933741595 epoch‚33 % _timestamp‚1.7843518543015828e+09 _runtime‚11512.994963378 0loss/FRt1/assignment_loss‚1.4859161281845614 3loss/SRqqt2/assignment_loss‚0.42028956953107255 3loss/FBt2/assignment_loss‚0.0010145717526650363 1loss/SRqqt1/detection_loss‚0.8046064173506403 /loss/FBt1/detection_loss‚0.3810493245043538 3loss/FBt1/assignment_loss‚0.0008371745815133638 /loss/FRt2/detection_loss‚1.3399017827444615 trainer/global_step‚13399 _step‚1104 2loss/SRqqt1/assignment_loss‚0.4952036662861081 /loss/FRt1/detection_loss‚0.5567168691789162€c¯* 0loss/FRt2/assignment_loss‚0.5591389108146703 1loss/SRqqt2/detection_loss‚0.5736395215673665 /loss/FBt2/detection_loss‚0.2805849972452658 %loss/total_loss‚6.898898933741595 epoch‚33 % _timestamp‚1.7843518543015828e+09 _runtime‚11512.994963378 0loss/FRt1/assignment_loss‚1.4859161281845614 3loss/SRqqt2/assignment_loss‚0.42028956953107255 3loss/FBt2/assignment_loss‚0.0010145717526650363 1loss/SRqqt1/detection_loss‚0.8046064173506403 /loss/FBt1/detection_loss‚0.3810493245043538 3loss/FBt1/assignment_loss‚0.0008371745815133638 /loss/FRt2/detection_loss‚1.3399017827444615 trainer/global_step‚13399 _step‚1104 2loss/SRqqt1/assignment_loss‚0.4952036662861081 /loss/FRt1/detection_loss‚0.5567168691789162ÐN¡Ý°*‚(:’ õ˜ìÒš¤” gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^¤Pmã±*‚(:Ø õ˜ìÒþ¿Î– network.recv‚ 3.095383e+06 disk.drbd2165.out‚0 disk.sda3.out‚ 339.7421875 proc.cpu.threads‚17 network.sent‚ 2.1373386e+07) disk./.usagePercent‚39.28185493556311% disk.nvme0c0n1.out‚30119.23046875 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 39.0791015625* proc.memory.percent‚16.989850997924805 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 18724.8125 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 18724.8125 disk.dm-0.out‚30119.23046875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 18724.8125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125$ memory_percent‚97.39296436309814$ disk./.usageGB‚585.3280258178711 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 339.7421875" proc.memory.rssMB‚ 2783.6171875 disk.dm-1.out‚ 339.7421875# disk.nvme0n1.out‚30119.23046875( proc.memory.availableMB‚ 427.13671875 cpu‚26.17667377711698X¸€Â²*‚(:’ „™ìÒχݓ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚244.313' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚48.86259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶·†$è³*‚(:Ý „™ìÒ똖$ memory_percent‚97.39012718200684 network.sent‚ 2.1436749e+07 network.recv‚ 3.100034e+06) disk./.usagePercent‚39.28187669620352 disk.dm-1.out‚ 340.0078125 cpu‚26.1763819988412 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 18724.8125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 340.0078125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 39.0791015625# proc.memory.rssMB‚ 2793.32421875 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 18724.8125" disk.dm-0.out‚30121.9345703125 disk.nvme0n1.in‚ 18724.8125' disk.nvme0c0n1.out‚30121.9345703125 disk.sda2.in‚0* proc.memory.percent‚17.049098014831543 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚30121.9345703125 disk.sda3.out‚ 340.0078125' proc.memory.availableMB‚ 427.6015625$ disk./.usageGB‚585.3283500671387 disk.sda1.in‚0 disk.sda2.out‚0wàËî‘´*‚(:† “™ìÒâïä“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Zöhîµ*‚(:ã “™ìÒôþé– disk.sda2.out‚0* disk./.usagePercent‚39.281866711909686 disk.nvme0n1.in‚ 18724.8125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 39.1416015625 proc.cpu.threads‚17 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚30126.74560546875 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 340.3984375 disk.rbd0.in‚0$ memory_percent‚97.39172458648682( proc.memory.availableMB‚ 427.33984375 network.sent‚ 2.1441825e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 18724.8125 disk.sda.out‚ 340.3984375# proc.memory.rssMB‚ 2793.32421875 cpu‚26.175999113691056 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0# disk.dm-0.out‚30126.74560546875 disk.sda2.in‚0* proc.memory.percent‚17.049098014831543$ disk./.usageGB‚585.3282012939453 disk.dm-1.out‚ 340.3984375 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 network.recv‚ 3.10371e+06 disk.dm-0.in‚ 18724.8125& disk.nvme0n1.out‚30126.74560546875¾šÄ4¶*/ _wandbruntime‚11553  _runtime‚11553‚«*BÌ·*Æ ' lr-AdamW/pg2‚0.00011282032765048994 trainer/global_step‚13449 % _timestamp‚1.7843518947217643e+09 _runtime‚11553.415556349 _step‚1105 ' lr-AdamW/pg1‚0.00011282032765048994y”×Ѹ*Ë trainer/global_step‚13449 % _timestamp‚1.7843518947217643e+09 _runtime‚11553.415556349 _step‚1105 ' lr-AdamW/pg1‚0.00011282032765048994 ' lr-AdamW/pg2‚0.00011282032765048994Ñðíó84¹*/ _wandbruntime‚11553  _runtime‚11553Ò+^(º*€P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464640954081792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14201176551471 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2058326248741376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784351894725482e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124575046 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14201517101361 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23559288 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2033589689520128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231201307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14223218124288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14375073708544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚11553.422699654 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124565069 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2044103388995584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14222876785152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450265880373248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106636238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2033589706559488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14222876785152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2058326248741376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2044103371956224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450265861761024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2058326607119872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106636238 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2047791223660849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚464640848989696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14201517101361 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23559288 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2033589689520128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231201307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14223218124288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14375073708544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚11553.422699654 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124565069 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2044103388995584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14222876785152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450265880373248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106636238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2033589706559488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14222876785152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2058326248741376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2044103371956224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450265861761024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1106 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2058326607119872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106636238 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2047791223660849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚464640848989696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2058326248741376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784351894725482e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124575046 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144ÒVÏW›4¼*/ _wandbruntime‚11554  _runtime‚11554!w'½*‹N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2058474860783616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124574121 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124574121 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044250612985856 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106643870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2033736200192000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2047938387797193 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14376092913664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14223889412096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2058474860783616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52466815 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14224230758400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14375998052352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚464675402045440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450299309131776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231217991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106643870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124584112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28906053 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14202528155195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28905782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2033736217231360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52467089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23561033 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1107 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2044250612985856šr'¾*N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14375998052352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚464675402045440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450299309131776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231217991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106643870 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124584112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28906053 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14202528155195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28905782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2033736217231360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52467089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23561033 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1107 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2044250612985856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2058474860783616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124574121 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124574121 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044250612985856 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106643870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2033736200192000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2047938387797193 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14376092913664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14223889412096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2058474860783616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52466815 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14224230758400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ó¢Ýe4¿*/ _wandbruntime‚11554  _runtime‚115542ÈkcÀ*Š % _timestamp‚1.7843518953456426e+09 0loss/FRt2/assignment_loss‚0.6351842809900193 1loss/SRqqt1/detection_loss‚0.7628787693477683 /loss/FBt2/detection_loss‚0.2672576247218617 _step‚1108 0loss/FRt1/assignment_loss‚1.4969754355141554 3loss/FBt1/assignment_loss‚6.281078416972489e-05 3loss/FBt2/assignment_loss‚1.090299847387794e-05 _runtime‚11554.038964904 2loss/SRqqt1/assignment_loss‚0.4122922048888889 /loss/FRt1/detection_loss‚0.5499777459410791 1loss/SRqqt2/detection_loss‚0.6341598355778948 &loss/total_loss‚7.1107942769485195 trainer/global_step‚13449 2loss/SRqqt2/assignment_loss‚0.4438774834749589 .loss/FRt2/detection_loss‚1.448262016879569 0loss/FBt1/detection_loss‚0.45985516582968067 epoch‚33Uøª9Á* 0loss/FRt1/assignment_loss‚1.4969754355141554 3loss/FBt1/assignment_loss‚6.281078416972489e-05 3loss/FBt2/assignment_loss‚1.090299847387794e-05 _runtime‚11554.038964904 2loss/SRqqt1/assignment_loss‚0.4122922048888889 /loss/FRt1/detection_loss‚0.5499777459410791 1loss/SRqqt2/detection_loss‚0.6341598355778948 &loss/total_loss‚7.1107942769485195 trainer/global_step‚13449 2loss/SRqqt2/assignment_loss‚0.4438774834749589 .loss/FRt2/detection_loss‚1.448262016879569 0loss/FBt1/detection_loss‚0.45985516582968067 epoch‚33 % _timestamp‚1.7843518953456426e+09 0loss/FRt2/assignment_loss‚0.6351842809900193 1loss/SRqqt1/detection_loss‚0.7628787693477683 /loss/FBt2/detection_loss‚0.2672576247218617 _step‚1108ÔÌË–®’Â*‚(:‡ ¢™ìÒߨŒ” gpu.0.gpu‚28.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãôTéÃ*‚(:Þ ¢™ìÒÛ”š–# disk./.usageGB‚585.328369140625 disk.drbd2165.in‚0 disk.sda3.out‚ 340.796875 disk.sda2.out‚0 disk.nvme0n1.in‚ 18724.8125 disk.sda1.out‚0( proc.memory.availableMB‚ 427.29296875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2793.32421875 disk.sda.in‚ 0.1328125 disk.sda.out‚ 340.796875 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚17.049098014831543 network.sent‚ 2.144685e+07 network.recv‚ 3.106936e+06) disk./.usagePercent‚39.28187797624119 disk.dm-1.out‚ 340.796875 disk.sda2.in‚0& disk.nvme0n1.out‚30129.72021484375 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 18724.8125( disk.nvme0c0n1.out‚30129.72021484375 disk.dm-0.in‚ 18724.8125 disk.sda3.in‚ 0.1328125$ memory_percent‚97.39201068878174 cpu‚26.175688040020887# disk.dm-0.out‚30129.72021484375 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 39.1416015625 disk.rbd0.in‚0 ÔÕ„žÄ*‚(:“ ±™ìÒÜÚÝ” gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶t³·ìÅ*‚(:á ±™ìÒÊßÒ•$ disk./.usageGB‚585.3272476196289 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2793.32421875 network.sent‚ 2.15115e+07' disk.nvme0c0n1.out‚30133.9677734375" disk.dm-0.out‚30133.9677734375 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 18794.3046875 disk.dm-1.in‚ 0.1328125 proc.cpu.threads‚17) disk./.usagePercent‚39.28180271002613 disk.sda2.out‚0% disk.nvme0n1.out‚30133.9677734375 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.out‚ 341.078125 disk.rbd1.in‚ 2.01953125 disk.dm-1.out‚ 341.078125( proc.memory.availableMB‚ 427.53515625* proc.memory.percent‚17.049098014831543 network.recv‚ 3.111254e+06# disk.nvme0c0n1.in‚ 18794.3046875 disk.sda3.out‚ 341.078125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 39.2041015625 cpu‚26.175420546816 disk.dm-0.in‚ 18794.3046875 disk.sda1.in‚0$ memory_percent‚97.39053249359131 disk.sda2.in‚0 disk.rbd0.in‚0R‚4Æ*/ _wandbruntime‚11594  _runtime‚11594P HšÊÇ*Ä & lr-AdamW/pg1‚0.0001112629680293567 & lr-AdamW/pg2‚0.0001112629680293567 trainer/global_step‚13499 % _timestamp‚1.7843519355278995e+09 _runtime‚11594.223446079 _step‚11095lÖØÏÈ*É trainer/global_step‚13499 % _timestamp‚1.7843519355278995e+09 _runtime‚11594.223446079 _step‚1109 & lr-AdamW/pg1‚0.0001112629680293567 & lr-AdamW/pg2‚0.0001112629680293567Õæëï4É*/ _wandbruntime‚11594  _runtime‚11594uhÎÊ*€P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107017840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52653233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125029498 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52652982 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065738995724800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29006692 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107017840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051465423437824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 % _timestamp‚1.7843519355307474e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2065738995724800 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107017838 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29006444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2051465423437824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚13499 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14273572286976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2051465440477184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23646541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚1110 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2065739354103296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14426065857024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107017838 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2051465440477184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2065739354103296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23646538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466363464345088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2055167373741357 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125029498 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14273913626112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14273572286976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14426146045440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2040915240153088 _runtime‚11594.224425328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125019521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚13499 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14273572286976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2051465440477184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23646541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚1110 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2065739354103296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14426065857024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107017838 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2051465440477184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2065739354103296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23646538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466363464345088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2055167373741357 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125029498 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14273913626112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14273572286976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14426146045440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2040915240153088 _runtime‚11594.224425328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125019521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065738995724800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29006692 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107017840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051465423437824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77Ö»0Ì»4Ì*/ _wandbruntime‚11594  _runtime‚11594ôL‡'Í*ŠN E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14274926260224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125038564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29008448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466397825684480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843519361470141e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23648286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23648283 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚11594.840806497 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107025472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2051612664467456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14427097238528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2065887607767040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52656998 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14427177419776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 trainer/global_step‚13499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2065887249381376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451970747058176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064036 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14252804092101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107025472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107025470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125028573 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14274584913920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29008712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚1111 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14274584913920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52656731 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2051612664467456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466397924477952ÃFo|›Î*N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064036 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14252804092101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107025472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107025470 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125028573 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14274584913920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29008712 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚1111 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14274584913920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52656731 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2051612664467456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466397924477952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14274926260224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125038564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29008448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466397825684480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843519361470141e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23648286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23648283 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚11594.840806497 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107025472 [FDeviceStatR¾þz sMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2051612664467456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14427097238528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2065887607767040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52656998 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14427177419776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 trainer/global_step‚13499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2065887249381376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451970747058176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 Q4Ï*/ _wandbruntime‚11594  _runtime‚11594ûÖmÐ*ˆ /loss/FRt1/assignment_loss‚1.490240166205931 2loss/SRqqt1/assignment_loss‚0.4650551879489735 /loss/FRt1/detection_loss‚0.5132702827471579 .loss/FRt2/detection_loss‚1.458752666809924 /loss/FBt1/detection_loss‚0.3898430014284508 trainer/global_step‚13499 4loss/FBt1/assignment_loss‚1.0251266117956177e-05 2loss/FBt2/assignment_loss‚0.004594439882173602 1loss/SRqqt1/detection_loss‚0.8920027381081649 1loss/SRqqt2/detection_loss‚0.5367238784980735 0loss/FBt2/detection_loss‚0.38576363076614184 %loss/total_loss‚7.125736649317348 epoch‚33 _runtime‚11594.844083443 0loss/FRt2/assignment_loss‚0.5395756852179067 3loss/SRqqt2/assignment_loss‚0.44990472043833235 $ _timestamp‚1.784351936150783e+09 _step‚1112ÜÏãÑ* /loss/FRt1/assignment_loss‚1.490240166205931 2loss/SRqqt1/assignment_loss‚0.4650551879489735 /loss/FRt1/detection_loss‚0.5132702827471579 .loss/FRt2/detection_loss‚1.458752666809924 /loss/FBt1/detection_loss‚0.3898430014284508 trainer/global_step‚13499 4loss/FBt1/assignment_loss‚1.0251266117956177e-05 2loss/FBt2/assignment_loss‚0.004594439882173602 1loss/SRqqt1/detection_loss‚0.8920027381081649 1loss/SRqqt2/detection_loss‚0.5367238784980735 0loss/FBt2/detection_loss‚0.38576363076614184 %loss/total_loss‚7.125736649317348 epoch‚33 _runtime‚11594.844083443 0loss/FRt2/assignment_loss‚0.5395756852179067 3loss/SRqqt2/assignment_loss‚0.44990472043833235 $ _timestamp‚1.784351936150783e+09 _step‚1112ØÚÊÖ®‘Ò*‚(:† À™ìÒ§í“ gpu.0.gpu‚81.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íAv¹õÓ*‚(:ê À™ìÒù뜗' proc.memory.availableMB‚ 427.2109375 disk.dm-0.in‚ 18794.3046875 network.sent‚ 2.1514565e+07 disk.dm-1.out‚ 341.26953125 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 18794.3046875 disk.sda3.in‚ 0.1328125* proc.memory.percent‚17.049098014831543 cpu‚26.17527291765781' disk.nvme0c0n1.out‚30136.6220703125% disk.nvme0n1.out‚30136.6220703125 disk.sda3.out‚ 341.26953125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.28182267861379" disk.dm-0.out‚30136.6220703125 disk.sda.out‚ 341.26953125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.39251136779785 disk.drbd2165.out‚0 disk.sda2.out‚0$ disk./.usageGB‚585.3275451660156 disk.rbd0.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18794.3046875 disk.rbd1.out‚ 39.2041015625# proc.memory.rssMB‚ 2793.32421875 network.recv‚ 3.114756e+06 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125­5*:’Ô*‚(:‡ Ï™ìÒ‰½Þ“ gpu.0.gpu‚10.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öbgÚñÕ*‚(:æ Ï™ìÒ…Ýø•' disk.nvme0c0n1.out‚30143.0263671875 disk.rbd0.in‚0 disk.dm-1.out‚ 341.7109375# memory_percent‚97.3928451538086 disk.rbd1.in‚ 2.01953125 disk.sda2.in‚0 network.sent‚ 2.1577791e+07 network.recv‚ 3.119346e+06# disk.nvme0c0n1.in‚ 18794.3046875 disk.sda3.in‚ 0.1328125 disk.sda.out‚ 341.7109375 disk.dm-1.in‚ 0.1328125% proc.memory.availableMB‚ 427.15625* proc.memory.percent‚17.049098014831543 disk.sda2.out‚0% disk.nvme0n1.out‚30143.0263671875 disk.rbd1.out‚ 39.2666015625 disk.rbd0.out‚0 cpu‚26.175203270345687 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚585.3274421691895 disk.dm-0.in‚ 18794.3046875 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2793.32421875" disk.dm-0.out‚30143.0263671875 proc.cpu.threads‚17* disk./.usagePercent‚39.281815766410375 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 18794.3046875 disk.sda3.out‚ 341.7109375ˆxºžÖ*‚(:“ Þ™ìÒ☆“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚255.879' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.175799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ƒçìî×*‚(:ã Þ™ìÒÚ–„–" disk.dm-0.out‚30146.8759765625 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0% proc.memory.availableMB‚ 427.15625* disk./.usagePercent‚39.281810646259686 disk.dm-1.out‚ 342.140625 disk.drbd2165.out‚0# memory_percent‚97.3928451538086 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 342.140625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18794.3046875# proc.memory.rssMB‚ 2793.32421875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 39.2666015625 network.recv‚ 3.122767e+06 disk.sda.in‚ 0.1328125 disk.sda.out‚ 342.140625 disk.sda2.out‚0! disk.nvme0n1.in‚ 18794.3046875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚30146.8759765625 cpu‚26.175106160639945 network.sent‚ 2.1582877e+07 disk.sda1.in‚0$ disk./.usageGB‚585.3273658752441 disk.dm-0.in‚ 18794.3046875% disk.nvme0n1.out‚30146.8759765625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚17.049098014831543èÀ£Á4Ø*/ _wandbruntime‚11635  _runtime‚11635< 'ËÙ*Å ' lr-AdamW/pg1‚0.00010971288388212774 ' lr-AdamW/pg2‚0.00010971288388212774 trainer/global_step‚13549 $ _timestamp‚1.784351976542756e+09 _runtime‚11635.236318895 _step‚1113A‹³ÐÚ*Ê ' lr-AdamW/pg1‚0.00010971288388212774 ' lr-AdamW/pg2‚0.00010971288388212774 trainer/global_step‚13549 $ _timestamp‚1.784351976542756e+09 _runtime‚11635.236318895 _step‚1113ÙÒ‘D4Û*/ _wandbruntime‚11635  _runtime‚116359ÊOâ(Ü*ÿO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 $ _timestamp‚1.784351976543833e+09 _step‚1114 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058827474919424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073151742708224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232873411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23733791 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107399438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125473973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125483950 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23733788 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52842894 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14324267788800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468085873403392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14324267788800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107399440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2062543523821865 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14324609127936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125483950 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2048240756707328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107399440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107399438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468085984786944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453608756226048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29109106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453608737613824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52843152 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232883390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2048240773746688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2058827491958784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14302409525287 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125473973 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14477135789568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2073152101086720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2073152101086720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2058827474919424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29109361 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2062543166232615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Áæ* (Ý*„P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107399438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125473973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125483950 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23733788 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52842894 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14324267788800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468085873403392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14324267788800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107399440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2062543523821865 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14324609127936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125483950 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2048240756707328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107399440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107399438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468085984786944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453608756226048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29109106 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453608737613824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52843152 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232883390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2048240773746688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2058827491958784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14302409525287 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125473973 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14477135789568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2073152101086720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2073152101086720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2058827474919424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29109361 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2062543166232615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 $ _timestamp‚1.784351976543833e+09 _step‚1114 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058827474919424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073151742708224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232873411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23733791ÚAã¶4Þ*/ _wandbruntime‚11635  _runtime‚11635ë½ÖØ ß*‹N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 232890095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52846620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468120434053120 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2048387267379200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107407070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058974732988416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073299996364800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232900088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107407072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚13549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125483025 GADeviceStÐ%9atsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107407070 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125483025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468120318482432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14303761129011 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14303420579009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125493016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2062691045547571 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843519771578681e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23735533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062690687958209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107407072 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2073300354750464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14478249068544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14478152110080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2073300354750464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888o– 'à*N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14303761129011 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14303420579009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125493016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2062691045547571 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843519771578681e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23735533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062690687958209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107407072 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2073300354750464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14478249068544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14478152110080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2073300354750464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 232890095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52846620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468120434053120 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2048387267379200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107407070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058974732988416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073299996364800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232900088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107407072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚13549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125483025 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107407070 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125483025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468120318482432Û«²ÃÒ4á*/ _wandbruntime‚11635  _runtime‚11635/àÄä â*… epoch‚33 2loss/SRqqt2/assignment_loss‚0.4452621267100373 2loss/FBt2/assignment_loss‚0.002049835665889173 /loss/FRt1/detection_loss‚0.5803594616904841 1loss/SRqqt2/detection_loss‚0.5783146506855726 /loss/FBt2/detection_loss‚0.4301503294506863 2loss/SRqqt1/assignment_loss‚0.5036739014423709 3loss/FBt1/assignment_loss‚0.0006250166166505043 1loss/SRqqt1/detection_loss‚0.7672212855244752 trainer/global_step‚13549 _step‚1116 /loss/FRt1/assignment_loss‚1.535130574879308 .loss/FRt2/detection_loss‚1.386934644269404 /loss/FBt1/detection_loss‚0.5163423199586472 $ _timestamp‚1.784351977161393e+09 _runtime‚11635.854692138 0loss/FRt2/assignment_loss‚0.5414837888658863 %loss/total_loss‚7.287547935759411˜¹ ã*Š 0loss/FRt2/assignment_loss‚0.5414837888658863 %loss/total_loss‚7.287547935759411 epoch‚33 2loss/SRqqt2/assignment_loss‚0.4452621267100373 2loss/FBt2/assignment_loss‚0.002049835665889173 /loss/FRt1/detection_loss‚0.5803594616904841 1loss/SRqqt2/detection_loss‚0.5783146506855726 /loss/FBt2/detection_loss‚0.4301503294506863 2loss/SRqqt1/assignment_loss‚0.5036739014423709 3loss/FBt1/assignment_loss‚0.0006250166166505043 1loss/SRqqt1/detection_loss‚0.7672212855244752 trainer/global_step‚13549 _step‚1116 /loss/FRt1/assignment_loss‚1.535130574879308 .loss/FRt2/detection_loss‚1.386934644269404 /loss/FBt1/detection_loss‚0.5163423199586472 $ _timestamp‚1.784351977161393e+09 _runtime‚11635.854692138Ü×ä*‚(:… í™ìÒÜï” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.65' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.13 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘k¶`òå*‚(:ç í™ìÒÆè®– disk.sda.in‚ 0.1328125 disk.sda.out‚ 342.4609375 disk.rbd1.out‚ 39.3916015625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.sda3.in‚ 0.1328125 network.sent‚ 2.1587791e+07* disk./.usagePercent‚39.281809878237084 disk.dm-1.out‚ 342.4609375 disk.dm-1.in‚ 0.1328125" disk.nvme0c0n1.in‚ 18877.765625 disk.dm-0.in‚ 18877.765625 disk.nvme0n1.in‚ 18877.765625 disk.sda3.out‚ 342.4609375* proc.memory.percent‚17.049098014831543 disk.sda1.out‚0" disk.dm-0.out‚30153.2041015625$ memory_percent‚97.39305973052979( proc.memory.availableMB‚ 427.12109375 network.recv‚ 3.126223e+06$ disk./.usageGB‚585.3273544311523 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚30153.2041015625% disk.nvme0n1.out‚30153.2041015625# proc.memory.rssMB‚ 2793.32421875 cpu‚26.174879429217476 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0q2Y“æ*‚(:ˆ ü™ìÒÉóí“ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áðç*‚(:å ü™ìÒŒ…™– network.recv‚ 3.130433e+06 disk.sda1.out‚0 disk.dm-0.in‚ 18877.765625 disk.nvme0n1.in‚ 18877.765625# proc.memory.rssMB‚ 2793.32421875 cpu‚26.174805450625392$ disk./.usageGB‚585.3275260925293 disk.rbd0.in‚0" disk.dm-0.out‚30155.9091796875 network.sent‚ 2.1650947e+07 disk.dm-1.out‚ 342.9296875$ memory_percent‚97.39232063293457 disk.sda1.in‚0% disk.nvme0n1.out‚30155.9091796875 disk.sda3.out‚ 342.9296875 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚17.049098014831543 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0) disk./.usagePercent‚39.28182139857612 disk.rbd0.out‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 427.2421875' disk.nvme0c0n1.out‚30155.9091796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 18877.765625 disk.sda.in‚ 0.1328125 disk.sda.out‚ 342.9296875 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 39.3916015625ö‹-ážè*‚(:“ ‹šìÒ”ÙÕ” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚236.492' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.298399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bš&Ûõé*‚(:ê ‹šìÒÊ‚¯– disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 18877.765625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 39.4541015625# proc.memory.rssMB‚ 2793.32421875 network.recv‚ 3.133925e+06 disk.sda1.out‚0# disk.dm-0.out‚30160.91748046875 disk.nvme0n1.in‚ 18877.765625$ memory_percent‚97.39232063293457* proc.memory.percent‚17.049098014831543 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 343.33984375 disk.sda2.out‚0& disk.nvme0n1.out‚30160.91748046875 disk.sda3.in‚ 0.1328125 proc.cpu.threads‚17) disk./.usagePercent‚39.28182472667407 disk.sda.in‚ 0.1328125( disk.nvme0c0n1.out‚30160.91748046875$ disk./.usageGB‚585.3275756835938 disk.rbd0.in‚0 disk.dm-1.out‚ 343.33984375 disk.dm-0.in‚ 18877.765625 disk.sda3.out‚ 343.33984375 cpu‚26.17473019350257 network.sent‚ 2.1656075e+07 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125' proc.memory.availableMB‚ 427.2421875ߦij4ê*/ _wandbruntime‚11676  _runtime‚11676­Àà`Ìë*Æ ' lr-AdamW/pg1‚0.00010817017476781138 ' lr-AdamW/pg2‚0.00010817017476781138 trainer/global_step‚13599 % _timestamp‚1.7843520173758533e+09 _runtime‚11676.069482624 _step‚1117†¡Ì¸Ñì*Ë ' lr-AdamW/pg1‚0.00010817017476781138 ' lr-AdamW/pg2‚0.00010817017476781138 trainer/global_step‚13599 % _timestamp‚1.7843520173758533e+09 _runtime‚11676.069482624 _step‚1117ÝåvBq4í*/ _wandbruntime‚11676  _runtime‚11676ŒRU$î*P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2066189526401024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14374963290624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23821038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455280194152448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14528087686144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2069919673902373 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2055566307340288 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469808263226368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23821041 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2080564489691648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2080564489691648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455280175540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233709463 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843520173770626e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125938402 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2066189543440384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53033069 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14374963290624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125938402 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2066189526401024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469808378804224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2080564848070144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14353366562085 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14375304629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29211758 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066189543440384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125928425 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2069919316313123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107781038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125928425 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2080564848070144 Êw²XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29212028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2055566290300928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14353026012195 trainer/global_step‚13599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107781040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14375304629760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169?Š€ò (ï*†P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2080564489691648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2080564489691648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455280175540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233709463 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843520173770626e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125938402 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2066189543440384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53033069 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14374963290624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125938402 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2066189526401024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469808378804224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2080564848070144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14353366562085 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14375304629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29211758 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066189543440384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125928425 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2069919316313123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107781038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125928425 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2080564848070144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29212028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2055566290300928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14353026012195 trainer/global_step‚13599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107781040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14375304629760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2066189526401024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14374963290624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23821038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455280194152448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14528087686144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2069919673902373 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2055566307340288 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469808263226368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23821041 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479ÞžÅÓ×4ð*/ _wandbruntime‚11676  _runtime‚11676J‰.'ñ*‹N WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2070067195628079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469842702886400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2070066838038717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2066336784470016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2080712743348224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2066336767430656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14375975917568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2066336767430656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53036872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455313622910976 $ _timestamp‚1.784352017992411e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233726147 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736140 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107788670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2066336784470016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14354037065917 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23822783 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14376317263872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080713101733888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2055712800972800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23822786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14354377615919 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚13599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107788672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107788672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53036585 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2055712818012160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14529098587648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107788670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14375975917568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29214086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÿg'ò*N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233726147 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736140 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107788670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2066336784470016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14354037065917 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23822783 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14376317263872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080713101733888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2055712800972800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23822786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14354377615919 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚13599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107788672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107788672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53036585 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2055712818012160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14529098587648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107788670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14375975917568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29214086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2070067195628079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469842702886400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2070066838038717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2066336784470016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2080712743348224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2066336767430656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14375975917568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2066336767430656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53036872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455313622910976 $ _timestamp‚1.784352017992411e+09ßj×áR4ó*/ _wandbruntime‚11676  _runtime‚11676‚7$ ô*‡ 2loss/SRqqt2/assignment_loss‚0.4693046960399451 /loss/FRt1/detection_loss‚0.5477650770834381 _step‚1120 0loss/FRt1/assignment_loss‚1.4791893104673812 2loss/SRqqt1/assignment_loss‚0.5023750436580764 1loss/FBt1/assignment_loss‚0.02392222912262508 /loss/FRt2/detection_loss‚1.3746362673950627 1loss/SRqqt2/detection_loss‚0.5970240701486856 /loss/FBt1/detection_loss‚0.6729030530156207 0loss/FBt2/detection_loss‚0.33681241594581657 epoch‚33 3loss/FBt2/assignment_loss‚0.0006340241010137956 %loss/total_loss‚7.567396995448293 trainer/global_step‚13599 _runtime‚11676.689376169 /loss/FRt2/assignment_loss‚0.643380294107411 1loss/SRqqt1/detection_loss‚0.9194505143632162 % _timestamp‚1.7843520179960523e+09, nõ*Œ /loss/FRt2/assignment_loss‚0.643380294107411 1loss/SRqqt1/detection_loss‚0.9194505143632162 % _timestamp‚1.7843520179960523e+09 2loss/SRqqt2/assignment_loss‚0.4693046960399451 /loss/FRt1/detection_loss‚0.5477650770834381 _step‚1120 0loss/FRt1/assignment_loss‚1.4791893104673812 2loss/SRqqt1/assignment_loss‚0.5023750436580764 1loss/FBt1/assignment_loss‚0.02392222912262508 /loss/FRt2/detection_loss‚1.3746362673950627 1loss/SRqqt2/detection_loss‚0.5970240701486856 /loss/FBt1/detection_loss‚0.6729030530156207 0loss/FBt2/detection_loss‚0.33681241594581657 epocz‹‚¤h‚33 3loss/FBt2/assignment_loss‚0.0006340241010137956 %loss/total_loss‚7.567396995448293 trainer/global_step‚13599 _runtime‚11676.689376169वoýö*‚(:„ ššìÒ䘸– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î–ÔKõ÷*‚(:ê ššìÒ…ä— disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 18877.765625 disk.sda2.out‚0( proc.memory.availableMB‚ 426.75390625 network.recv‚ 3.137484e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚30163.92236328125 disk.dm-0.in‚ 18877.765625 disk.nvme0n1.in‚ 18877.765625 proc.cpu.threads‚17# disk.dm-0.out‚30163.92236328125 disk.sda.out‚ 343.71484375& disk.nvme0n1.out‚30163.92236328125 disk.sda3.out‚ 343.71484375 disk.sda1.out‚0* proc.memory.percent‚17.053651809692383 disk.rbd1.out‚ 39.4541015625 disk.sda1.in‚0 disk.dm-1.out‚ 343.71484375 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚97.39530086517334 network.sent‚ 2.1673897e+07) disk./.usagePercent‚39.28182651872681 disk.rbd1.in‚ 2.01953125 cpu‚26.17635660202487$ disk./.usageGB‚585.3276023864746 disk.sda3.in‚ 0.1328125 disk.sda.in‚ 0.1328125" proc.memory.rssMB‚ 2794.0703125Ä·‡4ø*/ _wandbruntime‚11695  _runtime‚11695g!Ù$t=ù*îz :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36068716645240784 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7456068992614746 *jet/accuracy_3_of_3‚0.0440124049782753 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.565353274345398 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22038456797599792 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7748848795890808 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6636505126953125 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02928699180483818 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333094984292984 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5728699564933777 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10713425278663635 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3384418785572052 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8695766925811768 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997744619846344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994059145450592 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6446461081504822 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2442282885313034 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.684350311756134 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8588813543319702 jet/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24290990829467773 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6854058504104614 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_3_of_4‚0.042860183864831924 jet/accuracy_2_of_5‚1 (loss/val_total_loss‚7.61098110764804 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968851804733276 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5284494757652283 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07187888771295547 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15554405748844147 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28510984778404236 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22774307429790497 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9473532438278198 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 _step‚1121 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7957949042320251 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016454730182886124 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2355436384677887 _runtime‚11695.516740287 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7056439518928528 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6276028752326965 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7283987402915955 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28674405813217163 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5129472017288208 *jet/accuracy_1_of_3‚0.9182581901550293 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6065225601196289 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939308166503906 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6080254912376404 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2148377150297165 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7287623286247253 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.800753116607666 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6624044179916382 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843520368225415e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025083059445023537 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949723482131958 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04853774234652519 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033408842980861664 )jet/accuracy_2_of_3‚0.516602635383606 6validation_average_jet_accuracy‚0.3763585388660431 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21585816144943237 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9836512804031372 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21392790973186493 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23275041580200195 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5021868348121643 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878772497177124 +particle/specificity‚0.9013798832893372 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23088338971138 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6735793352127075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7105773091316223 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2116268426179886 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23109281063079834 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 (particle/accuracy‚0.8374488353729248 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3691651523113251 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967158436775208 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02889520302414894 +jet/accuracy_4_of_4‚0.01908157207071781 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958267211914062 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7585856914520264 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.795729398727417 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12502159178256989 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7922585606575012 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7143058776855469 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8712396025657654 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28509870171546936 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12158623337745667 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952384233474731 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7652226686477661 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2851141393184662 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107111811637878 +jet/accuracy_2_of_2‚0.13226349651813507 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024874499067664146 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7121447324752808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10596445947885513 epoch‚33 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7109425663948059 particle/accuracy_1_of_1‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034989017993211746 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23268751800060272 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.641581654548645 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.312738835811615 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5454166531562805 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6524795889854431 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968851804733276 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9503050446510315 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7631903290748596 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.309678852558136 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7724589109420776 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6160040497779846 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.626375675201416 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6336541175842285 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016454730182886124 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8814100623130798 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8712396025657654 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7900628447532654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09415565431118011 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7073885798454285 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.054587047547101974 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20941150188446045 )jet/accuracy_1_of_1‚0.343253493309021 'particle/f_score‚0.6146389245986938 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2956148684024811 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6776758432388306 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8814100623130798 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8878368139266968 +particle/sensitivity‚0.6036746501922607 trainer/global_step‚13599 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871999621391296 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7876681089401245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6273081302642822 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5042487978935242 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.642486572265625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.626375675201416 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3036687672138214 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2962440848350525 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13975933194160461 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977266192436218 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8899848461151123 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7823764085769653 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2768515646457672 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28510984778404236'—s<ú*óz @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7109425663948059 particle/accuracy_1_of_1‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034989017993211746 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23268751800060272 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.641581654548645 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.312738835811615 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5454166531562805 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6524795889854431 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968851804733276 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9503050446510315 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7631903290748596 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.309678852558136 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7724589109420776 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6160040497779846 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.626375675201416 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6336541175842285 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016454730182886124 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8814100623130798 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8712396025657654 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7900628447532654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09415565431118011 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7073885798454285 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.054587047547101974 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20941150188446045 )jet/accuracy_1_of_1‚0.343253493309021 'particle/f_score‚0.6146389245986938 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2956148684024811 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6776758432388306 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8814100623130798 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8878368139266968 +particle/sensitivity‚0.6036746501922607 trainer/global_step‚13599 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871999621391296 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7876681089401245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6273081302642822 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5042487978935242 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.642486572265625 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.626375675201416 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3036687672138214 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2962440848350525 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13975933194160461 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977266192436218 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8899848461151123 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7823764085769653 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2768515646457672 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28510984778404236 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36068716645240784 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7456068992614746 *jet/accuracy_3_of_3‚0.0440124049782753 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.565353274345398 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22038456797599792 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7748848795890808 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6636505126953125 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02928699180483818 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333094984292984 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5728699564933777 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10713425278663635 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3384418785572052 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8695766925811768 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997744619846344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994059145450592 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6446461081504822 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2442282885313034 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.684350311756134 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8588813543319702 jet/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24290990829467773 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6854058504104614 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_3_of_4‚0.042860183864831924 jet/accuracy_2_of_5‚1 (loss/val_total_loss‚7.61098110764804 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968851804733276 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5284494757652283 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07187888771295547 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15554405748844147 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28510984778404236 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22774307429790497 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9473532438278198 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 _step‚1121 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7957949042320251 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016454730182886124 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2355436384677887 _runtime‚11695.516740287 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7056439518928528 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6276028752326965 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7283987402915955 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28674405813217163 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5129472017288208 *jet/accuracy_1_of_3‚0.9182581901550293 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6065225601196289 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939308166503906 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6080254912376404 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2148377150297165 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7287623286247253 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.800753116607666 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6624044179916382 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843520368225415e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025083059445023537 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949723482131958 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04853774234652519 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033408842980861664 )jet/accuracy_2_of_3‚0.516602635383606 6validation_average_jet_accuracy‚0.3763585388660431 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21585816144943237 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9836512804031372 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21392790973186493 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23275041580200195 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5021868348121643 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9878772497177124 +particle/specificity‚0.9013798832893372 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23088338971138 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6735793352127075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7105773091316223 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2116268426179886 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23109281063079834 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 (particle/accuracy‚0.8374488353729248 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3691651523113251 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967158436775208 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02889520302414894 +jet/accuracy_4_of_4‚0.01908157207071781 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958267211914062 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7585856914520264 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.795729398727417 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12502159178256989 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7922585606575012 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7143058776855469 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8712396025657654 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28509870171546936 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12158623337745667 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952384233474731 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7652226686477661 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2851141393184662 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt@€?s1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7107111811637878 +jet/accuracy_2_of_2‚0.13226349651813507 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024874499067664146 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7121447324752808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10596445947885513 epoch‚33áâœû*‚(:‘ ©šìÒÇó“ gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÄjüü*‚(:ñ ©šìÒù¨ë• disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 344.05078125# disk.dm-0.out‚30169.17529296875* proc.memory.percent‚17.110013961791992 network.recv‚ 3.141964e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚18950.93359375 disk.sda.out‚ 344.05078125 network.sent‚ 2.1724058e+07$ disk./.usageGB‚585.3288116455078" disk.nvme0n1.in‚18950.93359375 disk.sda3.out‚ 344.05078125 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚18950.93359375 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.sda3.in‚ 0.1328125* disk./.usagePercent‚39.281907673115164& disk.nvme0n1.out‚30169.17529296875 disk.rbd1.out‚ 39.5166015625 disk.sda1.in‚0$ memory_percent‚97.45602607727051' proc.memory.availableMB‚ 416.8046875" proc.memory.rssMB‚ 2803.3046875 cpu‚26.178676568341025 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30169.17529296875 disk.sda.in‚ 0.1328125=B‡o“ý*‚(:ˆ ¸šìÒµþÝ“ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/˜öÔóþ*‚(:è ¸šìÒëÆø• disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125) disk./.usagePercent‚39.28189538475352$ disk.nvme0c0n1.in‚18950.93359375 disk.sda3.out‚ 344.34375 disk.sda2.out‚0& disk.nvme0n1.out‚30173.47998046875( disk.nvme0c0n1.out‚30173.47998046875 disk.dm-0.in‚18950.93359375# disk.dm-0.out‚30173.47998046875 disk.sda2.in‚0* proc.memory.percent‚17.110013961791992 cpu‚26.178469153585585 disk.rbd0.out‚0" proc.memory.rssMB‚ 2803.3046875 network.recv‚ 3.146294e+06 disk.sda.in‚ 0.1328125 disk.sda.out‚ 344.34375 disk.rbd1.out‚ 39.5166015625 disk.sda1.out‚0 disk.dm-1.out‚ 344.34375 disk.drbd2165.in‚0$ memory_percent‚97.45619297027588( proc.memory.availableMB‚ 416.77734375" disk.nvme0n1.in‚18950.93359375 network.sent‚ 2.1781505e+07$ disk./.usageGB‚585.3286285400391 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.drbd2165.out‚0‰Nr½“ÿ*‚(:ˆ ÇšìÒÆè“ gpu.0.gpu‚79.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚253.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÓåé€+‚(:Þ ÇšìÒ’œˆ– disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚18950.93359375 cpu‚26.178262240657023 network.sent‚ 2.1786613e+07) disk./.usagePercent‚39.28190152893434$ disk./.usageGB‚585.3287200927734$ disk.nvme0c0n1.in‚18950.93359375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 344.59375 network.recv‚ 3.149742e+06% disk.nvme0c0n1.out‚30178.52734375 disk.dm-0.out‚30178.52734375 disk.sda.out‚ 344.59375# disk.nvme0n1.out‚30178.52734375# memory_percent‚97.4562406539917( proc.memory.availableMB‚ 416.76953125" disk.nvme0n1.in‚18950.93359375 disk.dm-1.out‚ 344.59375" proc.memory.rssMB‚ 2803.3046875 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 39.5791015625* proc.memory.percent‚17.110013961791992 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0úülR4+/ _wandbruntime‚11736  _runtime‚11736¤aóÊ‚+Ä ' lr-AdamW/pg1‚0.00010663493977173134 ' lr-AdamW/pg2‚0.00010663493977173134 trainer/global_step‚13649 $ _timestamp‚1.784352077331887e+09 _runtime‚11736.02542396 _step‚1122èbä σ+É ' lr-AdamW/pg1‚0.00010663493977173134 ' lr-AdamW/pg2‚0.00010663493977173134 trainer/global_step‚13649 $ _timestamp‚1.784352077331887e+09 _runtime‚11736.02542396 _step‚1122â§ÍK4„+/ _wandbruntime‚11736  _runtime‚11736WÒU(…+€P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2075481161572352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29347739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843520773326793e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _runtime‚11736.026390594 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471565068983296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14421362224332 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108283304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53258807 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126463176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234746482 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108283306 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2064800504147968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126453199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471565163589632 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14443391997952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2064800487108608 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2079221849332940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14443391997952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2079222206922190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14421702774222 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2075481144532992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456967525566464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2089924536530944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23910811 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2l (†+…P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚13649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234736503 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108283304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2089924894909440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2075481161572352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14443733337088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2064800487108608 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2079221849332940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14443391997952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2079222206922190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14421702774222 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2075481144532992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456967525566464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2089924536530944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23910811 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2075481161572352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29347739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843520773326793e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _runtime‚11736.026390594 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471565068983296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14421362224332 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108283304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53258807 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126463176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234746482 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108283306 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2064800504147968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126453199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471565163589632 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14443391997952 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53262378 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753187 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2064947014819840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126462251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457000954324992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234753187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14444404624896 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14598574425600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23912553 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29350079 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14598650412544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843520779507236e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2090073148573184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2090073148573184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471599623349760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126472242 UEDeviceStatsMonitor.on_train_batch_e°ïË nd/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23912556 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126472242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2075628402601984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079369371058534 _runtime‚11736.64450955 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29349825 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14422373278054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2075628385562624 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2075628402601984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2090072790187520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2090073148573184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471599623349760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126472242 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23912556 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126472242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2075628402601984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079369371058534 _runtime‚11736.64450955 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29349825 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14422373278054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2075628385562624 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2075628402601984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2090072790187520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53262378 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753187 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2064947014819840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126462251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457000954324992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234753187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14444404624896 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14598574425600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23912553 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29350079 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14598650412544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0äÓm04Š+/ _wandbruntime‚11736  _runtime‚11736­žX ‹+‰ 3loss/FBt1/assignment_loss‚9.884397414635278e-07 2loss/FBt2/assignment_loss‚0.002687587050717766 1loss/SRqqt1/detection_loss‚0.7511629213154426 1loss/SRqqt2/detection_loss‚0.5951636040075703 0loss/FBt2/detection_loss‚0.23818587833883526 % _timestamp‚1.7843520779537737e+09 _step‚1125 &loss/total_loss‚6.6313009854915235 0loss/FBt1/detection_loss‚0.21315475477604195 epoch‚34 /loss/FRt1/assignment_loss‚1.497893012271208 0loss/FRt2/assignment_loss‚0.6091343344977269 3loss/SRqqt1/assignment_loss‚0.42172463741094796 /loss/FRt1/detection_loss‚0.5249464425250671 .loss/FRt2/detection_loss‚1.394706852961043 trainer/global_step‚13649 _runtime‚11736.647088278 1loss/SRqqt2/assignment_loss‚0.382539971897181A=^gŒ+Ž % _timestamp‚1.7843520779537737e+09 _step‚1125 &loss/total_loss‚6.6313009854915235 0loss/FBt1/detection_loss‚0.21315475477604195 epoch‚34 /loss/FRt1/assignment_loss‚1.497893012271208 0loss/FRt2/assignment_loss‚0.6091343344977269 3loss/SRqqt1/assignment_loss‚0.42172463741094796 /loss/FRt1/detection_loss‚0.5249464425250671 .loss/FRt2/detection_loss‚1.394706852961043 trainer/global_step‚13649 _runtime‚11736.647088278 1loss/SRqqt2/assignment_loss‚0.382539971897181 3loss/FBt1/assignment_loss‚9.884397414635278e-07 2loss/FBt2/assignment_loss‚0.002687587050717766 1loss/SRqqt1/detection_loss‚0.7511629213154426 1loss/SRqqt2/detection_loss‚0.5951636040075703 0loss/FBt2/detection_loss‚0.23818587833883526åš¡¤“+‚(:ˆ ÖšìÒö¯Ó“ gpu.0.gpu‚66.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚235.669' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÊÿéŽ+‚(:Þ ÖšìÒíœä– disk.dm-0.out‚ 30181.328125$ memory_percent‚97.45368957519531* disk./.usagePercent‚39.281843671231606" disk.nvme0n1.in‚18950.93359375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 344.85546875" proc.memory.rssMB‚ 2803.3046875 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 39.5791015625 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚18950.93359375 network.sent‚ 2.179153e+07$ disk./.usageGB‚585.3278579711914 disk.sda.out‚ 344.85546875 disk.drbd2165.out‚0# disk.nvme0c0n1.out‚ 30181.328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125 network.recv‚ 3.153298e+06 disk.sda3.out‚ 344.85546875 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 30181.328125 disk.drbd2165.in‚0$ proc.memory.availableMB‚417.1875* proc.memory.percent‚17.110013961791992 disk.rbd1.in‚ 2.01953125 disk.dm-0.in‚18950.93359375 disk.sda3.in‚ 0.1328125 cpu‚26.178010523863723˜“+‚(:ˆ åšìÒ­¦Ä“ gpu.0.gpu‚93.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02 ùNö+‚(:ë åšìÒ’ãÚ• disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 network.sent‚ 2.185474e+07 disk.sda2.out‚0" proc.memory.rssMB‚ 2803.3046875* proc.memory.percent‚17.110013961791992 disk.drbd2165.out‚0" disk.nvme0n1.in‚19018.89453125 network.recv‚ 3.157852e+06$ disk./.usageGB‚585.3281440734863# disk.dm-0.out‚30193.05615234375 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.45373725891113$ disk.nvme0c0n1.in‚19018.89453125( disk.nvme0c0n1.out‚30193.05615234375 disk.sda1.in‚0 disk.dm-1.out‚ 346.4140625' proc.memory.availableMB‚ 417.1796875 disk.drbd2165.in‚0 disk.sda.out‚ 346.4140625 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 346.4140625 disk.rbd1.out‚ 39.6416015625) disk./.usagePercent‚39.28186287179667 disk.dm-0.in‚19018.89453125 disk.rbd0.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚30193.05615234375 disk.dm-1.in‚ 0.1328125 cpu‚26.17768295012069['“‘+‚(:ˆ ôšìÒ˜Ž¸” gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ€Årö’+‚(:ë ôšìÒ¹¬ó• disk.sda3.in‚ 0.1328125 disk.sda1.out‚0# disk.dm-0.out‚30197.16650390625 disk.sda.out‚ 347.89453125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 39.6416015625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 347.89453125 cpu‚26.177413911687015 network.sent‚ 2.185971e+07 network.recv‚ 3.16127e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.28186261578914 disk.sda2.in‚0& disk.nvme0n1.out‚30197.16650390625$ disk.nvme0c0n1.in‚19018.89453125" proc.memory.rssMB‚ 2803.3046875$ disk./.usageGB‚585.3281402587891 disk.sda3.out‚ 347.89453125* proc.memory.percent‚17.110013961791992 disk.dm-0.in‚19018.89453125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚30197.16650390625$ memory_percent‚97.45521545410156$ proc.memory.availableMB‚416.9375 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚19018.89453125bJY04“+/ _wandbruntime‚11776  _runtime‚11776Ho[áÊ”+Ä & lr-AdamW/pg1‚0.0001051072774991624 & lr-AdamW/pg2‚0.0001051072774991624 trainer/global_step‚13699 % _timestamp‚1.7843521181567795e+09 _runtime‚11776.850440978 _step‚1126!E(·Ï•+É % _timestamp‚1.7843521181567795e+09 _runtime‚11776.850440978 _step‚1126 & lr-AdamW/pg1‚0.0001051072774991624 & lr-AdamW/pg2‚0.0001051072774991624 trainer/global_step‚13699æ?{~d4–+/ _wandbruntime‚11776  _runtime‚11776õ ¡(—+€P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126917628 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2082843196014592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473288126717440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473288021625344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2072126037741568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23998058 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2086598357002698 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29453121 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14649058132480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 $ _timestamp‚1.784352118157767e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14494428838912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14649144612352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108664906 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2082843213053952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14494428838912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458638963492864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2097337641892864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126907651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14472319261130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53450925 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2097337283514368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108664904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77ßÜÂë¾ ˜+…P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53450925 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2097337283514368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108664904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126917628 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2082843196014592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473288126717440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473288021625344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2072126037741568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23998058 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2086598357002698 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29453121 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14649058132480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 $ _timestamp‚1.784352118157767e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14494428838912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14649144612352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108664906 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2082843213053952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14494428838912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458638963492864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2097337641892864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126907651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14472319261130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2çЛÊp4™+/ _wandbruntime‚11777  _runtime‚11777SH•"'š+ŠN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473322589393408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14495100126720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2082990454083584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126926694 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126916703 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2097485537170944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53454661 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23999803 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2097485895556608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14650178529792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108672536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14473330314964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23999806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2072272548413440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14495441473024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843521187762225e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2082990454083584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2072272531374080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589239 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14472989764962 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589239 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2097485895556608¦ïeL'›+N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53454922 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458672410863616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53454661 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23999803 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2097485895556608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14650178529792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108672536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14473330314964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23999806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2072272548413440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14495441473024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843521187762225e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2082990454083584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2072272531374080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589239 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14472989764962 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589239 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2097485895556608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473322589393408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14495100126720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2082990454083584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126926694 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126916703 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2097485537170944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2104750030497792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2104750030497792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090205264535552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127362103 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14545124340736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚13749 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2090205264535552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2093974149493956 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14522935748038 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2090205247496192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843521592068696e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2090205247496192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475010936798208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14544783001600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236418586 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109046506 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14545124340736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127372080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475010842191872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127372080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109046504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29557780 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460310401419264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14700516766720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843521592068696e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2090205247496192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475010936798208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14544783001600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236418586 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109046506 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14545124340736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127372080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475010842191872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127372080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109046504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29557780 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460310401419264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14700516766720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2104750030497792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2104750030497792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090205264535552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127362103 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14545124340736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088ëÑj‹Ë4©+/ _wandbruntime‚11818  _runtime‚11818o\ç'ª+‹N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2104898642540032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚13749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109054138 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14545795628544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2094121671219550 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843521598193066e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24087056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29559619 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090352488525824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109054136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2094122028808912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236425291 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127371155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2104898284154368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14701465604608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435284 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127381146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475045396673024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127381146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53646934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24087053 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109054136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11818.513045526 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2079598064967680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53646672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090352505565184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460343848790016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127381146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53646934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24087053 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109054136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11818.513045526 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2079598064967680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53646672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090352505565184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460343848790016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Qplit.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460343830177792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2104898642540032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚13749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109054138 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14545795628544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2094121671219550 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843521598193066e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24087056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29559619 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090352488525824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109054136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2094122028808912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236425291 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127371155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2104898284154368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14701465604608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435284 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127381146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475045396673024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097567298977792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2112162777481216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14751947020288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24172558 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101350657163714 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚13799 _runtime‚11858.724501454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14751854248960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2112162777481216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109428106 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53833684 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461981857957888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29660866 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097567298977792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14595478503424 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127816555 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476733804978176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097567298977792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2112162777481216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14751947020288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24172558 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101350657163714 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚13799 _runtime‚11858.724501454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14751854248960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2112162777481216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109428106 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53833684 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461981857957888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29660866 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097567298977792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14595478503424 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127816555 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476733804978176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStat§^¼›œsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261343 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127825607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2112311031137792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53837160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2097714540007424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097714557046784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109435738 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097714557046784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109435738 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2101497821300058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2097714540007424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚462015286716416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24174306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476768254154240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14596832476672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462015268104192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101498178889420 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14596491130368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109435736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2086923615600640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14574563288780 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2112311389523456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29663108 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127825607 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2112311389523456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14752967437824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚13799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14574222738778 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843522006471388e+09 _runtime‚11859.34087819 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237271336 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2101497821300058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2097714540007424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚462015286716416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24174306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476768254154240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14596832476672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462015268104192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101498178889420 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14596491130368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109435736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2086923615600640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14574563288780 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2112311389523456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29663108 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127825607 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2112311389523456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14752967437824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚13799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14574222738778 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843522006471388e+09 _runtime‚11859.34087819 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237271336 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261343 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127825607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2112311031137792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53837160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2097714540007424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097714557046784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109435738 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097714557046784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109435738 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54024613 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478456576598016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119575524464640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2104929367498752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238090690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2104929350459392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2119575882843136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚11899.771434628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128280984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54024885 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29765074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29764805 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14646174005248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2094102638522368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14624168721854 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14803280713728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109809704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3“! (Ì+†P JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1139 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109809706 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2119575882843136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14803194233856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24259808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463653277272064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2108726449654972 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128271007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24259811 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚13849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54024613 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478456576598016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119575524464640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2104929367498752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238090690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2104929350459392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2119575882843136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚11899.771434628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128280984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54024885 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29765074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29764805 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14646174005248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2094102638522368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14624168721854 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14803280713728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109809704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14646515344384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463653295884288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088ó#ø–4Í+/ _wandbruntime‚11900  _runtime‚11900ÛMJ'Î+‹N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109817338 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238097395 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109817336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2108874328969928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24261556 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109817338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29767150 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2094249132154880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119724136506880 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54028455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108873971380566 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14624839225686 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2105076591489024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843522416932216e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128280059 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478491027592704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109817336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14804302949888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238097395 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14804218574336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24261553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0IEí©)Ï+N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109817338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29767150 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2094249132154880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119724136506880 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54028455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108873971380566 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14624839225686 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2105076591489024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843522416932216e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128280059 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478491027592704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109817336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14804302949888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238097395 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14804218574336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24261553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109817338 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238097395 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109817336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2108874328969928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24261556 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101ôXK04Ð+/ _wandbruntime‚11900  _runtime‚11900çmÒZ Ñ+… 0loss/FBt1/detection_loss‚0.18612746375649955 trainer/global_step‚13849 _step‚1141 /loss/FRt1/detection_loss‚0.5726698432641004 1loss/SRqqt2/detection_loss‚0.5308070706382922 0loss/FRt2/assignment_loss‚0.5369570450658187 2loss/FBt1/assignment_loss‚0.006818660764676662 /loss/FBt2/detection_loss‚0.4011952065160856 epoch‚34 % _timestamp‚1.7843522416969028e+09 _runtime‚11900.39022649 0loss/FRt1/assignment_loss‚1.4300390531223608 2loss/FBt2/assignment_loss‚0.002924111821632653 %loss/total_loss‚6.755131934657105 2loss/SRqqt1/assignment_loss‚0.4323936510853944 1loss/SRqqt2/assignment_loss‚0.449573397890201 .loss/FRt2/detection_loss‚1.381963745344125 1loss/SRqqt1/detection_loss‚0.8236626853879184’.rOÒ+Š /loss/FRt1/detection_loss‚0.5726698432641004 1loss/SRqqt2/detection_loss‚0.5308070706382922 0loss/FRt2/assignment_loss‚0.5369570450658187 2loss/FBt1/assignment_loss‚0.006818660764676662 /loss/FBt2/detection_loss‚0.4011952065160856 epoch‚34 % _timestamp‚1.7843522416969028e+09 _runtime‚11900.39022649 0loss/FRt1/assignment_loss‚1.4300390531223608 2loss/FBt2/assignment_loss‚0.002924111821632653 %loss/total_loss‚6.755131934657105 2loss/SRqqt1/assignment_loss‚0.4323936510853944 1loss/SRqqt2/assignment_loss‚0.449573397890201 .loss/FRt2/detection_loss‚1.381963745344125 1loss/SRqqt1/detection_loss‚0.8236626853879184 0loss/FBt1/detection_loss‚0.18612746375649955 trainer/global_step‚13849 _step‚1141õðÚ–=“Ó+‚(:ˆ û›ìÒŽª’” gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýð¶“îÔ+‚(:ã û›ìÒóàž•# disk.dm-0.out‚30243.89599609375 disk.nvme0n1.in‚ 19338.5625& disk.nvme0n1.out‚30243.89599609375$ memory_percent‚97.46160507202148 disk.rbd1.out‚ 39.9541015625 disk.drbd2165.in‚0& proc.memory.availableMB‚ 415.890625 network.sent‚ 2.2079529e+07 disk.nvme0c0n1.in‚ 19338.5625 disk.rbd1.in‚ 2.01953125 disk.sda3.out‚ 353.91796875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 353.91796875 network.recv‚ 3.195387e+06 disk.sda1.in‚0 cpu‚26.175724895585486 disk.sda1.out‚0) disk./.usagePercent‚39.28188540045968( disk.nvme0c0n1.out‚30243.89599609375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 353.91796875 disk.sda2.in‚0* proc.memory.percent‚17.110013961791992 proc.cpu.threads‚17 disk.dm-0.in‚ 19338.5625 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2803.3046875$ disk./.usageGB‚585.3284797668457›)Ar‘Õ+‚(:† ŠœìÒÔ¸Ù” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æD¬±ÝÖ+‚(:Ò ŠœìÒ•ôÊ• disk.sda3.out‚ 354.21875 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0* disk./.usagePercent‚39.281898200836395# disk./.usageGB‚585.328670501709$ disk.nvme0n1.out‚30247.662109375* proc.memory.percent‚17.110037803649902 disk.sda.out‚ 354.21875 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.46017456054688 network.sent‚ 2.2142736e+07 disk.nvme0n1.in‚ 19338.5625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 19338.5625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 354.21875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 19338.5625 network.recv‚ 3.199725e+06! disk.dm-0.out‚30247.662109375 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.availableMB‚416.125 cpu‚26.175377327129194 disk.rbd1.out‚ 39.9541015625& disk.nvme0c0n1.out‚30247.662109375# proc.memory.rssMB‚ 2803.30859375aˆÕž×+‚(:“ ™œìÒóûê“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÊÎçØ+‚(:Ü ™œìҸѮ– disk.dm-1.in‚ 0.1328125% disk.nvme0c0n1.out‚30253.66015625 disk.sda.out‚ 354.45703125# disk.nvme0n1.out‚30253.66015625 disk.sda3.out‚ 354.45703125 disk.dm-0.out‚30253.66015625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.dm-1.out‚ 354.45703125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 19338.5625 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 40.0166015625$ memory_percent‚97.46108055114746' proc.memory.availableMB‚ 415.9765625* proc.memory.percent‚17.110037803649902$ disk./.usageGB‚585.3288078308105 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 19338.5625 cpu‚26.17523750566317 network.recv‚ 3.203331e+06* disk./.usagePercent‚39.281907417107625 disk.drbd2165.in‚0 disk.dm-0.in‚ 19338.5625# proc.memory.rssMB‚ 2803.30859375 network.sent‚ 2.2147758e+07m!%4Ù+/ _wandbruntime‚11940  _runtime‚11940Ø”‘ŽÊÚ+Ä _runtime‚11940.583348394 _step‚1142 & lr-AdamW/pg1‚9.907431059401262e-05 & lr-AdamW/pg2‚9.907431059401262e-05 trainer/global_step‚13899 % _timestamp‚1.7843522818898034e+09šB³mÏÛ+É _runtime‚11940.583348394 _step‚1142 & lr-AdamW/pg1‚9.907431059401262e-05 & lr-AdamW/pg2‚9.907431059401262e-05 trainer/global_step‚13899 % _timestamp‚1.7843522818898034e+09öO\q4Ü+/ _wandbruntime‚11940  _runtime‚11940Ì‚š(Ý+P E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480179200198656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14854485000192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465324715198464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14697210846208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2101428172115968 trainer/global_step‚13899 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚11940.584433973 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128735436 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2116102599735480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2126988629826560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2112291418980352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465324733810688 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2116102957324730 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112291418980352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14696869507072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128735436 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843522818905804e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29868895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24347061 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128725459 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54215953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110191304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110191306 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2126988271448064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2112291401940992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616—™ðË (Þ+†P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238926742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480179307387904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2126988271448064 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2126988629826560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29869161 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14674444658872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128725459 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54215953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110191304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110191306 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2126988271448064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2112291401940992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480179200198656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14854485000192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465324715198464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14697210846208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2101428172115968 trainer/global_step‚13899 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚11940.584433973 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128735436 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2116102599735480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2126988629826560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2112291418980352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465324733810688 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2116102957324730 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112291418980352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14696869507072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128735436 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843522818905804e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29868895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24347061 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128734511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54219952 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2101574682787840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14697882134016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train1ë¼£‡_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54219678 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480213648542208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2127136883490304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2127136883490304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2127136525104640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2101574665748480 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14698223480320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112438642970624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116250479050436 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14675796262596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24348806 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465358143956992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943440 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238933447 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110198936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843522825049758e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24348803 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112438660009984 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110198938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚11941.198752751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116250121461074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110198936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14855504585216w|À'á+N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110198936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14855504585216 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2112438660009984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128734511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54219952 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2101574682787840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14697882134016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54219678 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480213648542208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2127136883490304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2127136883490304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2127136525104640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2101574665748480 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14698223480320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112438642970624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116250479050436 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14675796262596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24348806 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465358143956992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943440 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238933447 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110198936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843522825049758e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24348803 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112438660009984 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110198938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚11941.198752751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116250121461074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ø>M‘4â+/ _wandbruntime‚11941  _runtime‚11941õ$¢ã+ /loss/FBt1/detection_loss‚0.4105341298744516 % _timestamp‚1.7843522825081096e+09 _runtime‚11941.201428251 3loss/SRqqt2/assignment_loss‚0.47387832533771657 /loss/FRt2/detection_loss‚1.4385493356986165 _step‚1145 0loss/FRt1/assignment_loss‚1.5305323446932293 4loss/FBt1/assignment_loss‚2.4581281878034854e-06 4loss/FBt2/assignment_loss‚0.00039743592841792643 1loss/SRqqt2/detection_loss‚0.6246128739855401 trainer/global_step‚13899 2loss/SRqqt1/assignment_loss‚0.5059283338709585 0loss/FBt2/detection_loss‚0.41406711027476767 &loss/total_loss‚7.3877046028066005 epoch‚34 0loss/FRt2/assignment_loss‚0.6146348453283861 /loss/FRt1/detection_loss‚0.5139290664103464 0loss/SRqqt1/detection_loss‚0.860638343275983VSòä+’ 0loss/FRt1/assignment_loss‚1.5305323446932293 4loss/FBt1/assignment_loss‚2.4581281878034854e-06 4loss/FBt2/assignment_loss‚0.00039743592841792643 1loss/SRqqt2/detection_loss‚0.6246128739855401 trainer/global_step‚13899 2loss/SRqqt1/assignment_loss‚0.5059283338709585 0loss/FBt2/detection_loss‚0.41406711027476767 &loss/total_loss‚7.3877046028066005 epoch‚34 0loss/FRt2/assignment_loss‚0.6146348453283861 /loss/FRt1/detection_loss‚0.5139290664103464 0loss/SRqqt1/detection_loss‚0.860638343275983 /loss/FBt1/detection_loss‚0.4105341298744516 % _timestamp‚1.7843522825081096e+09 _runtime‚11941.201428251 3loss/SRqqt2/assignment_loss‚0.47387832533771657 /loss/FRt2/detection_loss‚1.4385493356986165 _step‚1145ùElm‘å+‚(:† ¨œìÒ›Õ“ gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.142 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú T÷æ+‚(:ì ¨œìÒÒÁЗ( disk.nvme0c0n1.out‚30256.93017578125 disk.sda.in‚ 0.1328125 network.recv‚ 3.206889e+06 disk.sda3.out‚ 355.12109375 disk.rbd0.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 415.9609375# proc.memory.rssMB‚ 2803.30859375( disk./.usagePercent‚39.2819012729268 disk.dm-1.out‚ 355.12109375 disk.sda2.out‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 19442.7109375& disk.nvme0n1.out‚30256.93017578125 disk.dm-1.in‚ 0.1328125 network.sent‚ 2.2152811e+07 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.110037803649902 cpu‚26.175012477166618$ disk./.usageGB‚585.3287162780762 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0# disk.dm-0.out‚30256.93017578125 disk.sda.out‚ 355.12109375 disk.sda2.in‚0 disk.rbd1.out‚ 40.0166015625 disk.rbd0.out‚0 disk.dm-0.in‚ 19442.7109375# memory_percent‚97.4611759185791 disk.rbd1.in‚ 2.01953125# disk.nvme0c0n1.in‚ 19442.7109375z O“ç+‚(:ˆ ·œìÒÕÓæ” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÌ„0éè+‚(:Þ ·œìÒ Üæ•! disk.dm-0.out‚30262.326171875# disk.nvme0c0n1.in‚ 19442.7109375 disk.sda1.out‚0$ disk./.usageGB‚585.3287124633789 disk.sda3.out‚ 355.65625 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 19442.7109375 disk.sda1.in‚0$ memory_percent‚97.46119976043701* proc.memory.percent‚17.110037803649902 proc.cpu.threads‚17 network.recv‚ 3.210966e+06 disk.rbd1.out‚ 40.1416015625 disk.sda2.in‚0 cpu‚26.1749269849352) disk./.usagePercent‚39.28190101691927 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.out‚ 355.65625# proc.memory.rssMB‚ 2803.30859375( proc.memory.availableMB‚ 415.95703125 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚30262.326171875 network.sent‚ 2.2215164e+07 disk.dm-1.out‚ 355.65625 disk.dm-0.in‚ 19442.7109375 disk.sda.in‚ 0.1328125$ disk.nvme0n1.out‚30262.326171875&·º4é+/ _wandbruntime‚11981  _runtime‚11981ƒ”ì@Èê+ _runtime‚11981.537321029 _step‚1146 % lr-AdamW/pg1‚9.75859737748562e-05 % lr-AdamW/pg2‚9.75859737748562e-05 trainer/global_step‚13949 % _timestamp‚1.7843523228437595e+09\+ŽÍë+Ç _step‚1146 % lr-AdamW/pg1‚9.75859737748562e-05 % lr-AdamW/pg2‚9.75859737748562e-05 trainer/global_step‚13949 % _timestamp‚1.7843523228437595e+09 _runtime‚11981.537321029úFî4ì+/ _wandbruntime‚11981  _runtime‚11981«8U¬¡'í+P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2119653470461952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2108753705709568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14905787421696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2119653453422592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14747565008896 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14747906348032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2134401376809984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54406276 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129189888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119653453422592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2108753688670208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2119653470461952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129189888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2123478749815988 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14725061145780 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239762794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2134401376809984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2134401018431488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481901849872384 _runtime‚11981.538557784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24434311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24434308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29971968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2134401018431488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110572906 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110572904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29972233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚466996153124864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54406544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725401695670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1147 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocatƒx5fed‚14747906348032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129179911Ip6 (î+†P MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110572906 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110572904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29972233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚466996153124864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54406544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725401695670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1147 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14747906348032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129179911 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2119653470461952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2108753705709568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14905787421696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2119653453422592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14747565008896 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14747906348032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2134401376809984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54406276 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129189888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119653453422592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2108753688670208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2119653470461952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129189888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2123478749815988 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14725061145780 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239762794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2134401376809984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2134401018431488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481901849872384 _runtime‚11981.538557784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24434311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24434308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29971968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2134401018431488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ûëÖµ‰4ï+/ _wandbruntime‚11982  _runtime‚11982³¶«ü'ð+‹N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14748918982144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54410129 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2134549272088064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129198954 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129188963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24436056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2108900216381440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110580538 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54410401 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2119800711491584 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119800694452224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129198954 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11982.152969951 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110580536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481936405033472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2119800694452224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481936293657088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14726412749504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843523234592345e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29974345 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29974076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467029600495616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2134549630473728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123626271541582ÙüÐ4'ñ+N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843523234592345e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29974345 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29974076 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467029600495616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2134549630473728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123626271541582 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14748918982144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54410129 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2134549272088064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129198954 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129188963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24436056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2108900216381440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110580538 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54410401 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2119800711491584 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119800694452224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129198954 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11982.152969951 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110580536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481936405033472 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2119800694452224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481936293657088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14726412749504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169üe$/4ò+/ _wandbruntime‚11982  _runtime‚11982?nbP ó+† 0loss/FBt1/detection_loss‚0.21372977210007407 _runtime‚11982.155854897 0loss/FRt1/assignment_loss‚1.4213291357484783 2loss/SRqqt1/assignment_loss‚0.5268201204159645 3loss/SRqqt2/assignment_loss‚0.48507625308533875 2loss/FBt2/assignment_loss‚0.005584539341476782 0loss/SRqqt1/detection_loss‚0.857029582724846 0loss/FBt2/detection_loss‚0.38686172702610716 _step‚1149 /loss/FRt2/assignment_loss‚0.635768460375922 .loss/FRt1/detection_loss‚0.531909719472046 .loss/FRt2/detection_loss‚1.458268385794532 1loss/SRqqt2/detection_loss‚0.5755913258267855 epoch‚34 trainer/global_step‚13949 # _timestamp‚1.78435232346255e+09 4loss/FBt1/assignment_loss‚5.2160289348996005e-06 %loss/total_loss‚7.097974237940506a ™ô+‹ /loss/FRt2/assignment_loss‚0.635768460375922 .loss/FRt1/detection_loss‚0.531909719472046 .loss/FRt2/detection_loss‚1.458268385794532 1loss/SRqqt2/detection_loss‚0.5755913258267855 epoch‚34 trainer/global_step‚13949 # _timestamp‚1.78435232346255e+09 4loss/FBt1/assignment_loss‚5.2160289348996005e-06 %loss/total_loss‚7.097974237940506 0loss/FBt1/detection_loss‚0.21372977210007407 _runtime‚11982.155854897 0loss/FRt1/assignment_loss‚1.4213291357484783 2loss/SRqqt1/assignment_loss‚0.5268201204159645 3loss/SRqqt2/assignment_loss‚0.48507625308533875 2loss/FBt2/assignment_loss‚0.005584539341476782 0loss/SRqqt1/detection_loss‚0.857029582724846 0loss/FBt2/detection_loss‚0.38686172702610716 _step‚1149ýÆ}/F’õ+‚(:‡ ÆœìÒ´Ÿú” gpu.0.gpu‚24.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øh ö+‚(:î ÆœìÒ¢µÐ– disk.sda2.in‚0 disk.rbd1.out‚ 40.1416015625( disk.nvme0c0n1.out‚30265.40283203125 network.sent‚ 2.222004e+07 disk.dm-1.out‚ 356.06640625 disk.sda.out‚ 356.06640625 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.0195¸ï3125* proc.memory.percent‚17.110037803649902 disk.dm-1.in‚ 0.1328125 cpu‚26.174892094975853 disk.drbd2165.out‚0# disk.dm-0.out‚30265.40283203125 disk.rbd0.in‚0& disk.nvme0n1.out‚30265.40283203125# disk.nvme0c0n1.in‚ 19442.7109375 disk.dm-0.in‚ 19442.7109375$ disk./.usageGB‚585.3288650512695 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 356.06640625 disk.rbd0.out‚0( proc.memory.availableMB‚ 415.97265625 network.recv‚ 3.214282e+06) disk./.usagePercent‚39.28191125722064 disk.drbd2165.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2803.30859375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 19442.7109375$ memory_percent‚97.46110439300537\Ò‡“÷+‚(:ˆ ÕœìÒ‚Îû“ gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kp(ðø+‚(:å ÕœìÒꂼ–$ disk./.usageGB‚585.3290061950684 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 356.609375 cpu‚26.174716254092406 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 356.609375 disk.rbd1.out‚ 40.2041015625 network.sent‚ 2.2283178e+07) disk./.usagePercent‚39.28192072949941" disk.dm-0.out‚30271.9599609375 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0# proc.memory.rssMB‚ 2803.30859375 disk.sda.in‚ 0.1328125% disk.nvme0n1.out‚30271.9599609375 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 19442.7109375' disk.nvme0c0n1.out‚30271.9599609375 proc.cpu.threads‚17 disk.dm-0.in‚ 19442.7109375 disk.dm-1.in‚ 0.1328125' proc.memory.availableMB‚ 415.9296875* proc.memory.percent‚17.110037803649902 disk.sda.out‚ 356.609375 disk.rbd0.in‚0$ memory_percent‚97.46136665344238 network.recv‚ 3.219357e+06! disk.nvme0n1.in‚ 19442.7109375 disk.rbd0.out‚0—Íò:žù+‚(:“ äœìÒ¥ÄÁ“ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.843' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.568599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¬Óôú+‚(:é äœìÒçõŸ– disk.rbd1.out‚ 40.2041015625 disk.dm-1.out‚ 356.8828125# proc.memory.rssMB‚ 2803.30859375 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125* proc.memory.percent‚17.110037803649902 cpu‚26.174437377489305 disk.dm-0.in‚ 19611.9921875 disk.sda.in‚ 0.1328125! disk.nvme0n1.in‚ 19611.9921875 disk.rbd0.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 415.921875 network.recv‚ 3.222444e+06# disk.nvme0c0n1.in‚ 19611.9921875# disk.dm-0.out‚30276.61474609375& disk.nvme0n1.out‚30276.61474609375 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0) disk./.usagePercent‚39.28191996147681( disk.nvme0c0n1.out‚30276.61474609375 disk.sda.out‚ 356.8828125 disk.sda2.out‚0 network.sent‚ 2.2288206e+07 proc.cpu.threads‚17$ disk./.usageGB‚585.3289947509766 disk.sda2.in‚0 disk.sda3.out‚ 356.8828125 disk.rbd0.out‚0 disk.drbd2165.in‚0# memory_percent‚97.4614143371582o†±4û+/ _wandbruntime‚12022  _runtime‚12022Û`ä™Çü+Á % lr-AdamW/pg1‚9.61057908773875e-05 % lr-AdamW/pg2‚9.61057908773875e-05 trainer/global_step‚13999 % _timestamp‚1.7843523638617666e+09 _runtime‚12022.55847954 _step‚1150HÚðÌý+Æ trainer/global_step‚13999 % _timestamp‚1.7843523638617666e+09 _runtime‚12022.55847954 _step‚1150 % lr-AdamW/pg1‚9.61057908773875e-05 % lr-AdamW/pg2‚9.61057908773875e-05þ2·3å4þ+/ _wandbruntime‚12022  _runtime‚12022ðº(ÿ+€P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2141813765414912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30075912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2141814123793408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110954504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14798601849856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130854899896496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14956984575488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30075668 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54597226 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 $ _timestamp‚1.784352363865742e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129634363 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2141814123793408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2127015521943552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240598846 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129634363 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14798260510720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110954504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468667609663488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1151 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14956900192768 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110954506 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129644340 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2141814123793408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110954504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14798601849856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130854899896496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14956984575488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30075668 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54597226 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 $ _timestamp‚1.784352363865742e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129634363 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2141814123793408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2127015521943552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240598846 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129634363 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14798260510720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110954504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468667609663488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1151 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14956900192768 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110954506 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129644340 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2141813765414912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30075912 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ÿi>²4,/ _wandbruntime‚12023  _runtime‚12023U„ca‚,‹N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110962136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14957925835776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2131002779211452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843523644828682e+09 _step‚1152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240605551 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127162762973184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14958012308480 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14799614483968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30077900 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚13999 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2141962377457152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2141962019071488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2116225732935680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129643415 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129653406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468701038422016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116225749975040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14799273137664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127162762973184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54600941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24523303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483658945645568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2131002421622090 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14799273137664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_çΰbatch_end/active.all.allocated‚ 240615544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54601206 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14776688686410 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438†Íåb'ƒ,N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2141962019071488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2116225732935680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129643415 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129653406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468701038422016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116225749975040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14799273137664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127162762973184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54600941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24523303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483658945645568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2131002421622090 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14799273137664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240615544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54601206 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14776688686410 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110962136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14957925835776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2131002779211452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843523644828682e+09 _step‚1152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240605551 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127162762973184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14958012308480 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14799614483968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30077900 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚13999 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2141962377457152€ 4"æ4„,/ _wandbruntime‚12023  _runtime‚12023òLÑ …,Š /loss/FRt2/detection_loss‚1.3300465380922213 /loss/FBt2/detection_loss‚0.3265198735409381 _step‚1153 0loss/FBt1/detection_loss‚0.41812144186155037 %loss/total_loss‚6.971692371672148 epoch‚34 trainer/global_step‚13999 _runtime‚12023.178965426 0loss/FRt1/assignment_loss‚1.4399531905386702 0loss/FRt2/assignment_loss‚0.5516706363342148 3loss/SRqqt1/assignment_loss‚0.46457688345285486 1loss/SRqqt2/assignment_loss‚0.413394216019911 3loss/FBt1/assignment_loss‚4.365450043690481e-06 3loss/FBt2/assignment_loss‚0.0016838670944069615 /loss/FRt1/detection_loss‚0.5469646451832542 1loss/SRqqt1/detection_loss‚0.8455424301924561 1loss/SRqqt2/detection_loss‚0.6332142839116272 % _timestamp‚1.7843523644856904e+09Là™†, 3loss/FBt1/assignment_loss‚4.365450043690481e-06 3loss/FBt2/assignment_loss‚0.0016838670944069615 /loss/FRt1/detection_loss‚0.5469646451832542 1loss/SRqqt1/detection_loss‚0.8455424301924561 1loss/SRqqt2/detection_loss‚0.6332142839116272 % _timestamp‚1.7843523644856904e+09 /loss/FRt2/detection_loss‚1.3300465380922213 /loss/FBt2/detection_loss‚0.3265198735409381 _step‚1153 0loss/FBt1/detection_loss‚0.41812144186155037 %loss/total_loss‚6.971692371672148 epoch‚34 trainer/global_step‚13999 _runtime‚12023.178965426 0loss/FRt1/assignment_loss‚1.4399531905386702 0loss/FRt2/assignment_loss‚0.5516706363342148 3loss/SRqqt1/assignment_loss‚0.46457688345285486 1loss/SRqqt2/assignment_loss‚0.413394216019911 ›0Ý“‡,‚(:ˆ óœìÒÀŸŸ” gpu.0.gpu‚77.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…zÂ}ùˆ,‚(:î óœìÒø‘–! disk.nvme0n1.in‚ 19611.9921875 disk.rbd1.out‚ 40.2666015625 disk.dm-0.in‚ 19611.9921875# disk.nvme0c0n1.in‚ 19611.9921875 disk.sda2.in‚0# proc.memory.rssMB‚ 2805.94140625* proc.memory.percent‚17.126107215881348) disk./.usagePercent‚39.28191996147681 disk.sda.out‚ 357.16796875& disk.nvme0n1.out‚30283.83837890625 disk.sda1.in‚0 disk.dm-1.out‚ 357.16796875( disk.nvme0c0n1.out‚30283.83837890625 proc.cpu.threads‚17 disk.sda2.out‚0' proc.memory.availableMB‚ 413.4140625 network.sent‚ 2.2293184e+07 network.recv‚ 3.225846e+06$ disk./.usageGB‚585.3289947509766 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 357.16796875 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚30283.83837890625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0$ memory_percent‚97.47672080993652 cpu‚26.175609964104115 disk.sda.in‚ 0.1328125ã {4‰,/ _wandbruntime‚12041  _runtime‚12041‘Lykv=Š,ðz & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21700841188430786 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988316297531128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1942191869020462 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8817837834358215 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23709701001644135 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32531654834747314 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2807033061981201 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7463073134422302 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.13116852939128876 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982470273971558 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8934050798416138 )loss/val_total_loss‚7.617821210634558 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8817459344863892 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6319783329963684 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6586570739746094 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5191977620124817 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6910529136657715 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6157717704772949 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022585174068808556 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6512030959129333 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022585174068808556 +particle/sensitivity‚0.6083277463912964 &particle/f_score‚0.618785560131073 _step‚1154 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.792992115020752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5185185074806213 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7097765207290649 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7255240678787231 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8731288909912109 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247254014015198 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31485649943351746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3040789067745209 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7348560094833374 +jet/accuracy_4_of_4‚0.01670061983168125 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12843039631843567 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23572991788387299 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5144117474555969 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11052192747592926 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22923517227172852 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10304407775402069 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.602414071559906 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 /particle/accuracy_3_of_4‚0.9965986609458923 _runtime‚12041.857577958 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3497830331325531 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7682885527610779 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950152635574341 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6355674862861633 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09217556565999985 jet/accuracy_4_of_5‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30486029386520386 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2417377233505249 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 % _timestamp‚1.7843523831632597e+09 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3150750696659088 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05371087044477463 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22498823702335358 *jet/accuracy_2_of_4‚0.7576497197151184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960422515869141 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5296601057052612 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5022122263908386 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6230670213699341 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 *jet/accuracy_1_of_4‚0.9778740406036377 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31086793541908264 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2957894504070282 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487579941749573 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2932710647583008 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023153815418481827 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7188377976417542 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7189223766326904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8731288909912109 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8909255266189575 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022919299080967903 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0679173544049263 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966590404510498 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.559583306312561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6379602551460266 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6361119151115417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9475697875022888 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9021448493003845 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6361119151115417 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_1_of_4‚1 epoch‚34 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23130933940410614 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 jet/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2287113070487976 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7875846028327942 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7269126176834106 5validation_average_jet_accuracy‚0.384753555059433 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6950041651725769 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3658392131328583 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37553924322128296 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 +jet/accuracy_3_of_4‚0.04502468928694725 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9543709754943848 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.73548823595047 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8993574976921082 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965565800666809 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6519854068756104 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.671335756778717 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.153733029961586 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3502747118473053 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7929538488388062 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8390492796897888 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5747051239013672 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8138699531555176 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30486029386520386 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_1‚0.3540734648704529 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8028628826141357 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7899088859558105 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7953850030899048 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_3‚0.9219719767570496 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5123652219772339 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5706518292427063 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9061223864555359 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30609479546546936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6714209914207458 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7740358710289001 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.048202577978372574 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25884175300598145 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24761177599430084 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7613600492477417 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7176680564880371 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24515005946159363 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12584902346134186 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022007327526807785 trainer/global_step‚13999 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2569357454776764 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14421992003917694 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022337988018989563 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_3_of_3‚0.044085051864385605 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6651556491851807 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7786145806312561 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035420652478933334 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6930401921272278 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2216459959745407 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03675549104809761 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25A^We‹,õz 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022337988018989563 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_3_of_3‚0.044085051864385605 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6651556491851807 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7786145806312561 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035420652478933334 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6930401921272278 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2216459959745407 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03675549104809761 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21700841188430786 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988316297531128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1942191869020462 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8817837834358215 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23709701001644135 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32531654834747314 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/eÁ…‘6vent_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2807033061981201 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7463073134422302 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.13116852939128876 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982470273971558 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8934050798416138 )loss/val_total_loss‚7.617821210634558 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8817459344863892 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6319783329963684 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6586570739746094 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5191977620124817 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6910529136657715 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6157717704772949 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022585174068808556 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6512030959129333 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022585174068808556 +particle/sensitivity‚0.6083277463912964 &particle/f_score‚0.618785560131073 _step‚1154 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.792992115020752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5185185074806213 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7097765207290649 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7255240678787231 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8731288909912109 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247254014015198 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31485649943351746 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3040789067745209 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7348560094833374 +jet/accuracy_4_of_4‚0.01670061983168125 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12843039631843567 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23572991788387299 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5144117474555969 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11052192747592926 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22923517227172852 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10304407775402069 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.602414071559906 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 /particle/accuracy_3_of_4‚0.9965986609458923 _runtime‚12041.857577958 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3497830331325531 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7682885527610779 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950152635574341 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6355674862861633 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09217556565999985 jet/accuracy_4_of_5‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30486029386520386 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2417377233505249 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 % _timestamp‚1.7843523831632597e+09 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3150750696659088 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05371087044477463 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22498823702335358 *jet/accuracy_2_of_4‚0.7576497197151184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960422515869141 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5296601057052612 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5022122263908386 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6230670213699341 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 *jet/accuracy_1_of_4‚0.9778740406036377 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31086793541908264 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2957894504070282 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487579941749573 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2932710647583008 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023153815418481827 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7188377976417542 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7189223766326904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8731288909912109 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8909255266189575 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022919299080967903 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0679173544049263 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966590404510498 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.559583306312561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6379602551460266 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6361119151115417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9475697875022888 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9021448493003845 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6361119151115417 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_1_of_4‚1 epoch‚34 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23130933940410614 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 jet/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2287113070487976 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7875846028327942 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7269126176834106 5validation_average_jet_accuracy‚0.384753555059433 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6950041651725769 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3658392131328583 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37553924322128296 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 +jet/accuracy_3_of_4‚0.04502468928694725 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9543709754943848 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.73548823595047 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8993574976921082 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965565800666809 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6519854068756104 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.671335756778717 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.153733029961586 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3502747118473053 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7929538488388062 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8390492796897888 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5747051239013672 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8138699531555176 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30486029386520386 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_1‚0.3540734648704529 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8028628826141357 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7899088859558105 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7953850030899048 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_3‚0.9219719767570496 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5123652219772339 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5706518292427063 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9061223864555359 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30609479546546936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6714209914207458 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7740358710289001 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.048202577978372574 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25884175300598145 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24761177599430084 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7613600492477417 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7176680564880371 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24515005946159363 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12584902346134186 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022007327526807785 trainer/global_step‚13999 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2569357454776764 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14421992003917694 ("Purity/0FRt2SRqqt1FBt/event_purity‚0‚ ¥ß0ËŒ,‚(:’ ‚ìÒÉÍ£” gpu.0.gpu‚66.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚211.706' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚42.34119999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀU“Më,‚(:à ‚ìÒ§ô—$ disk./.usageGB‚585.3290748596191 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.49951362609863 cpu‚26.178079141988555' disk.nvme0c0n1.out‚30287.9658203125" disk.dm-0.out‚30287.9658203125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 357.40625! proc.memory.rssMB‚ 2809.890625 proc.cpu.threads‚17 disk.dm-0.in‚ 19611.9921875! disk.nvme0n1.in‚ 19611.9921875* proc.memory.percent‚17.150211334228516 network.recv‚ 3.230388e+06 disk.rbd0.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚30287.9658203125 disk.sda3.out‚ 357.40625 network.sent‚ 2.2356541e+07 disk.rbd1.out‚ 40.2666015625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 409.6796875 disk.dm-1.out‚ 357.40625 disk.sda2.out‚0) disk./.usagePercent‚39.28192533763502 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 19611.9921875 disk.sda.in‚ 0.1328125yz&ë“Ž,‚(:ˆ ‘ìÒߟݓ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.511' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¯ö,‚(:ë ‘ìÒï’¯• disk.sda2.in‚0( disk.nvme0c0n1.out‚30294.68408203125 disk.sda3.out‚ 357.58203125 proc.cpu.threads‚17( proc.memory.availableMB‚ 409.75390625# disk.nvme0c0n1.in‚ 19611.9921875# disk.dm-0.out‚30294.68408203125 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 357.58203125* proc.memory.percent‚17.150211334228516 cpu‚26.17784539527198$ memory_percent‚97.49906063079834 disk.dm-0.in‚ 19611.9921875 disk.rbd1.out‚ 40.3291015625 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.2414099e+07) disk./.usagePercent‚39.28191074520557# disk./.usageGB‚585.328857421875 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 19611.9921875 disk.rbd1.in‚ 2.01953125 disk.sda.out‚ 357.58203125 network.recv‚ 3.235099e+06 disk.rbd0.in‚0! proc.memory.rssMB‚ 2809.890625& disk.nvme0n1.out‚30294.68408203125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0/  “,‚(:ˆ  ìÒº¿Ú• gpu.0.gpu‚42.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊŸô¤ð‘,‚(:å  ìÒƒëð— disk.sda3.out‚ 357.87890625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0# memory_percent‚97.5006103515625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125% disk.nvme0n1.out‚30298.6435546875 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.out‚ 357.87890625 disk.drbd2165.in‚0* proc.memory.percent‚17.150211334228516) disk./.usagePercent‚39.28197858720215 disk.sda.out‚ 357.87890625 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19764.91015625 network.sent‚ 2.2419016e+07 network.recv‚ 3.238455e+06 disk.dm-0.in‚19764.91015625' disk.nvme0c0n1.out‚30298.6435546875! proc.memory.rssMB‚ 2809.890625" disk.dm-0.out‚30298.6435546875" disk.nvme0n1.in‚19764.91015625! proc.memory.availableMB‚409.5 cpu‚26.177613681234217$ disk./.usageGB‚585.3298683166504 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 40.3291015625d´Ù§4’,/ _wandbruntime‚12082  _runtime‚12082Êö'*É“,à & lr-AdamW/pg1‚9.463385697098816e-05 & lr-AdamW/pg2‚9.463385697098816e-05 trainer/global_step‚14049 % _timestamp‚1.7843524236607904e+09 _runtime‚12082.35967518 _step‚1155ÐhuôΔ,È & lr-AdamW/pg2‚9.463385697098816e-05 trainer/global_step‚14049 % _timestamp‚1.7843524236607904e+09 _runtime‚12082.35967518 _step‚1155 & lr-AdamW/pg1‚9.463385697098816e-05ƒ ^dò¼4•,/ _wandbruntime‚12082  _runtime‚12082Ê(–,€P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14844354394715 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2151174170632704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2136307140075520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14867030557184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130169114 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2136307123036160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2136307140075520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130159137 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14866689218048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125313436110848 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125313419071488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625886 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485381101241344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15026244546560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241615907 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2151173812254208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30208645 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111456770 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2151173812254208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14866689218048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2151174170632704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54819710 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470354941077504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784352423670506e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111456772 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470354959689728 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14867030557184 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2140157790505563 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24611331 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24611328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15026160163840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚12082.364467827 _step‚1156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111456772 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54819976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130169114 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2140157432916313 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088¸¼nóö—,…P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚14049 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130159137 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485381204236288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15026160163840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚12082.364467827 _step‚1156 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111456772 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54819976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130169114 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2140157432916313 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14844354394715 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2151174170632704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2136307140075520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14867030557184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130169114 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2136307123036160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2136307140075520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130159137 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14866689218048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚êÌ0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125313436110848 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125313419071488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625886 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485381101241344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15026244546560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241615907 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2151173812254208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30208645 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111456770 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2151173812254208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14866689218048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2151174170632704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54819710 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470354941077504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784352423670506e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111456772 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470354959689728 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14867030557184 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2140157790505563 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24611331 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24611328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2151322424296448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485415658262016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111464402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15027185438208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111464402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2136454364065792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 $ _timestamp‚1.784352424283905e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241632591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241642584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130168189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470388369836032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2140305312231269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125459929743360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2151322424296448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24613076 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30210450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470388388448256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚14049 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54823523 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14868043191296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2136454381105152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2151322065910784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130178180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30210698 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2136454381105152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24613073 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485415555274240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2136454364065792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111464404 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111464404 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54823523 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14868043191296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2136454381105152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2151322065910784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130178180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30210698 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2136454381105152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24613073 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485415555274240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2136454364065792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111464404 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111464404 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2151322424296448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485415658262016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111464402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15027185438208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111464402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2136454364065792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 $ _timestamp‚1.784352424283905e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241632591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241642584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130168189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470388369836032 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2140305312231269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125459929743360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2151322424296448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24613076 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30210450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470388388448256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚14049… *Ä¥Š4›,/ _wandbruntime‚12082  _runtime‚12082‰fÎ œ,† epoch‚35 0loss/FRt1/assignment_loss‚1.4899122417843678 2loss/SRqqt2/assignment_loss‚0.3980264982108617 2loss/FBt1/assignment_loss‚0.000563163776243177 2loss/FBt2/assignment_loss‚0.004043672214097084 _step‚1158 0loss/FBt1/detection_loss‚0.49815024093550164 _runtime‚12082.980384123 2loss/SRqqt1/assignment_loss‚0.5153693871626721 trainer/global_step‚14049 % _timestamp‚1.7843524242870731e+09 /loss/FRt2/assignment_loss‚0.583633282549305 /loss/FRt1/detection_loss‚0.5151264298858972 /loss/FRt2/detection_loss‚1.3184013300480695 1loss/SRqqt1/detection_loss‚0.8294935796121579 1loss/SRqqt2/detection_loss‚0.5580273702828628 /loss/FBt2/detection_loss‚0.3545011949450744 $loss/total_loss‚7.06524839140711§sx?,‹ /loss/FRt2/assignment_loss‚0.583633282549305 /loss/FRt1/detection_loss‚0.5151264298858972 /loss/FRt2/detection_loss‚1.3184013300480695 1loss/SRqqt1/detection_loss‚0.8294935796121579 1loss/SRqqt2/detection_loss‚0.5580273702828628 /loss/FBt2/detection_loss‚0.3545011949450744 $loss/total_loss‚7.06524839140711 epoch‚35 0loss/FRt1/assignment_loss‚1.4899122417843678 2loss/SRqqt2/assignment_loss‚0.3980264982108617 2loss/FBt1/assignment_loss‚0.000563163776243177 2loss/FBt2/assignment_loss‚0.004043672214097084 _step‚1158 0loss/FBt1/detection_loss‚0.49815024093550164 _runtime‚12082.980384123 2loss/SRqqt1/assignment_loss‚0.5153693871626721 trainer/global_step‚14049 % _timestamp‚1.7843524242870731e+09† 0v4“ž,‚(:ˆ ¯ìÒ›¼þ“ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óú½÷Ÿ,‚(:ì ¯ìÒñ³ý• disk.sda1.out‚0& disk.nvme0n1.out‚30305.15966796875* proc.memory.percent‚17.150211334228516 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚19764.91015625( disk.nvme0c0n1.out‚30305.15966796875 network.recv‚ 3.242058e+06 disk.drbd2165.out‚0 disk.dm-0.in‚19764.91015625# disk.dm-0.out‚30305.15966796875 disk.rbd1.out‚ 40.3916015625$ disk./.usageGB‚585.3300437927246 disk.rbd0.in‚0 disk.dm-1.out‚ 358.3515625 disk.drbd2165.in‚0) disk./.usagePercent‚39.28199036354872 disk.sda1.in‚0 disk.sda.out‚ 358.3515625 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚19764.91015625 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125( proc.memory.availableMB‚ 409.17578125! proc.memory.rssMB‚ 2809.890625 cpu‚26.17734053062317 network.sent‚ 2.2424046e+07$ memory_percent‚97.50258922576904 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 358.3515625¢†þ_“ ,‚(:ˆ ¾ìÒê¼ý” gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚278.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö\`õû¡,‚(:ð ¾ìҌ髖 disk.sda.out‚ 358.69140625 disk.sda2.out‚0' proc.memory.availableMB‚ 409.4140625" disk.nvme0n1.in‚19764.91015625 disk.dm-1.out‚ 358.69140625 disk.drbd2165.in‚0* proc.memory.percent‚17.150211334228516 network.sent‚ 2.2487125e+07$ disk./.usageGB‚585.3300552368164 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30308.08837890625 disk.sda2.in‚0* disk./.usagePercent‚39.281991131571324 disk.sda3.out‚ 358.69140625 proc.cpu.threads‚17 disk.sda1.in‚0# disk.dm-0.out‚30308.08837890625 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚30308.08837890625 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 network.recv‚ 3.246469e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚19764.91015625 disk.dm-0.in‚19764.91015625$ memory_percent‚97.50113487243652! proc.memory.rssMB‚ 2809.890625 cpu‚26.177057815104785 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 40.3916015625 disk.dm-1.in‚ 0.1328125÷ÍË‘¢,‚(:† ÍìÒß±à“ gpu.0.gpu‚72.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ´µ5÷£,‚(:ì ÍìÒŽ‹‡–$ disk.nvme0c0n1.in‚19764.91015625) disk./.usagePercent‚39.28199522769187 disk.sda1.out‚0 disk.sda.out‚ 358.94921875 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.dm-0.in‚19764.91015625% proc.memory.availableMB‚ 409.34375! proc.memory.rssMB‚ 2809.890625 cpu‚26.176922831775627 disk.sda2.out‚0" disk.nvme0n1.in‚19764.91015625 disk.rbd1.out‚ 40.4541015625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0# disk.dm-0.out‚30314.68603515625# memory_percent‚97.5015640258789* proc.memory.percent‚17.150211334228516 network.recv‚ 3.250236e+06$ disk./.usageGB‚585.3301162719727( disk.nvme0c0n1.out‚30314.68603515625& disk.nvme0n1.out‚30314.68603515625 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.dm-1.out‚ 358.94921875 proc.cpu.threads‚17 network.sent‚ 2.2492147e+07 disk.sda3.out‚ 358.94921875 disk.sda1.in‚0 disk.drbd2165.out‚0!¡@W4¤,/ _wandbruntime‚12123  _runtime‚12123ªžv…É¥,à & lr-AdamW/pg1‚9.317026659522255e-05 & lr-AdamW/pg2‚9.317026659522255e-05 trainer/global_step‚14099 % _timestamp‚1.7843524644498034e+09 _runtime‚12123.14346936 _step‚1159Î*Ü[4¦,È % _timestamp‚1.7843524644498034e+09 _ruÀäfâšntime‚12123.14346936 _step‚1159 & lr-AdamW/pg1‚9.317026659522255e-05 & lr-AdamW/pg2‚9.317026659522255e-05 trainer/global_step‚14099‡ Õ³rÒ4§,/ _wandbruntime‚12123  _runtime‚12123µz›H(¨,€P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111838370 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158586917616128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130613589 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472026397616128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2158586559237632 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2143669174517760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14917384719872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2143669191557120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158586559237632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14917726059008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30312813 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2147533582996821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2158586917616128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24698578 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472026379003904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2132638952665088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14894630331733 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111838372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872N> (©,…P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24698581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487103869064192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14894970881623 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242461938 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 _runtime‚12123.14450781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30312543 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2147533582996821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2158586917616128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24698578 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472026379003904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2132638952665088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14894630331733 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111838372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111838370 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158586917616128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130613589 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472026397616128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2158586559237632 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2143669174517760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14917384719872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2143669191557120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158586559237632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14917726059008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30312813 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130632632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30314594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487138313844224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12123.763362974 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2158735171279872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487138429414912 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843524650694711e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130632632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2143816415547392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2132785463336960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111846004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55014917 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472059807762432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15078603040256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242468643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1161 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111846002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2132785480376320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14895641385455 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130622641 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ýB—í«¬,N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487138429414912 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843524650694711e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130632632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2143816415547392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2132785463336960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111846004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55014917 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472059807762432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15078603040256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242468643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1161 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111846002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2132785480376320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14895641385455 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130622641 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14918738693120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14918397346816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55015184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130632632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30314594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487138313844224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12123.763362974 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2158735171279872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544‰ oÑDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2151031225999360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131068041 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15128555307520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2165999306221056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55202280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30416171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488826489912832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2154909733077329 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2165999306221056 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2151031225999360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488826372237824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2165999664599552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2151031243038720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112219972 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297990 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843525054938173e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288011 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14968080221696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2154910090666579 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473697816930304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14149 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112219970 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131068041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112219970 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2151031225999360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131068041 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15128555307520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2165999306221056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55202280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30416171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488826489912832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2154909733077329 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2165999306221056 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2151031225999360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488826372237824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2165999664599552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2151031243038720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112219972 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297990 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843525054938173e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288011 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14968080221696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2154910090666579 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473697816930304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14149 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112219970 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131068041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112219970 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55205796 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2155057254802923 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30418493 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488860932447232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473731245688832 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112227604 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14946257872363 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2151178484068352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15129573284864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131087084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14969434194944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843525061102023e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14946598422365 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112227602 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473731264301056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15129668146176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243304695 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2140111013969920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2151178484068352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2151178467028992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2151178467028992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2166147559877632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131077093 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112227602 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14969092848640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243304695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2166147918263296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112227604 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24787573 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2166147918263296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112227604 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24787573 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55205796 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2155057254802923 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30418493 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488860932447232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473731245688832 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112227604 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14946257872363 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2151178484068352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15129573284864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131087084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14969434194944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843525061102023e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14946598422365 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112227602 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473731264301056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15129668146176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243304695 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2140111013969920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2151178484068352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2151178467028992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2151178467028992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2166147559877632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131077093 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112227602 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14969092848640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243304695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24873081 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚475369273468928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14996203855439 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490548794335232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚12205.287587972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2173412053204480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15179636444160 _step‚1168 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2158393294520320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112601570 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2173412053204480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131532470 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocaóêÇ—8tions.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131522493 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244134042 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131522493 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55392521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244124063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2158393277480960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112601572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2173412411582976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15018775723520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30519714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112601570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475369254856704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15179539478528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490548909913088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2162285883157837 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15019117062656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30519443 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162286240747087 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2147290019852288 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2173412411582976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14995863305549 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438%ÜK¢ (Ë,†P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162286240747087 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2147290019852288 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2173412411582976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14995863305549 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843525465938663e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158393294520320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112601572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131532470 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚14199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15019117062656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24873081 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚475369273468928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14996203855439 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490548794335232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚12205.287587972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2173412053204480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15179636444160 _step‚1168 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2158393294520320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112601570 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2173412053204480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131532470 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131522493 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244134042 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131522493 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55392521 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244124063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2158393277480960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112601572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2173412411582976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15018775723520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30519714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112601570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475369254856704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15179539478528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490548909913088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2162285883157837 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15019117062656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30519443 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 sª¡»4Ì,/ _wandbruntime‚12205  _runtime‚12205iµ{/'Í,‹N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15180565060096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2162433762472793 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚14199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55396550 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14997214909273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131531545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158540535549952 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24874823 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30521472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15019788350464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112609202 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2147436547563520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162433404883431 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30521724 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244140747 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112609204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55396295 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2158540535549952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112609204 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475402702227456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843525472066739e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244150740 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158540518510592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15180651532800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ŽÝ2ýTÎ,N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14996874359271 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15020129696768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490583248675328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158540518510592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15019788350464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490583353760256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475402683615232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2173560665246720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2147436547563520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162433404883431 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30521724 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244140747 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112609204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55396295 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2158540535549952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112609204 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475402702227456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843525472066739e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244150740 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158540518510592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15180651532800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15180565060096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2162433762472793 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚14199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55396550 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚1499721490®³õT 9273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131531545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158540535549952 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24874823 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30521472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15019788350464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112609202 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F4Ï,/ _wandbruntime‚12205  _runtime‚12205gï6=Ð,‹ 0loss/FRt2/assignment_loss‚0.6291130115544743 3loss/SRqqt2/assignment_loss‚0.41680753392333336 1loss/SRqqt2/detection_loss‚0.5641193333980569 /loss/FBt1/detection_loss‚0.5296467650473157 trainer/global_step‚14199 % _timestamp‚1.7843525472109146e+09 _runtime‚12205.904268403 0loss/FRt1/assignment_loss‚1.4652457995471557 3loss/SRqqt1/assignment_loss‚0.38220844181713043 _step‚1170 3loss/FBt1/assignment_loss‚0.0006734867187335662 3loss/FBt2/assignment_loss‚0.0003691798207306312 /loss/FRt1/detection_loss‚0.5423568867974398 0loss/FBt2/detection_loss‚0.42090269362724037 epoch‚35 /loss/FRt2/detection_loss‚1.4418778375172188 0loss/SRqqt1/detection_loss‚0.830962963685348 %loss/total_loss‚7.224283933454178]ý-‘Ñ, 0loss/SRqqt1/detection_loss‚0.830962963685348 %loss/total_loss‚7.224283933454178 0loss/FRt2/assignment_loss‚0.6291130115544743 3loss/SRqqt2/assignment_loss‚0.41680753392333336 1loss/SRqqt2/detection_loss‚0.5641193333980569 /loss/FBt1/detection_loss‚0.5296467650473157 trainer/global_step‚14199 % _timestamp‚1.7843525472109146e+09 _runtime‚12205.904268403 0loss/FRt1/assignment_loss‚1.4652457995471557 3loss/SRqqt1/assignment_loss‚0.38220844181713043 _step‚1170 3loss/FBt1/assignment_loss‚0.0006734867187335662 3loss/FBt2/assignment_loss‚0.0003691798207306312 /loss/FRt1/detection_loss‚0.5423568867974398 0loss/FBt2/detection_loss‚0.42090269362724037 epoch‚35 /loss/FRt2/detection_loss‚1.4418778375172188’ ÑáF¯“Ò,‚(:ˆ §žìÒîÂŽ” gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z“¤ôÓ,‚(:é §žìÒ«‘Ê•* disk./.usagePercent‚39.288729761887595 disk.sda.in‚ 0.1328125 disk.sda2.out‚0% disk.nvme0n1.out‚30356.9716796875 cpu‚26.175726026576513 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚97.50337600708008$ disk./.usageGB‚585.4304656982422 disk.sda1.in‚0 disk.rbd1.out‚ 40.6416015625& proc.memory.availableMB‚ 409.046875! proc.memory.rssMB‚ 2809.890625* proc.memory.percent‚17.150211334228516 proc.cpu.threads‚17 network.sent‚ 2.2638269e+07 disk.sda1.out‚0 disk.dm-1.out‚ 365.8046875 disk.sda3.out‚ 365.8046875' disk.nvme0c0n1.out‚30356.9716796875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.out‚ 365.8046875" disk.dm-0.out‚30356.9716796875" disk.nvme0n1.in‚20178.98046875 disk.sda3.in‚ 0.1328125$ disk.nvme0c0n1.in‚20178.98046875 network.recv‚ 3.272688e+06 disk.dm-0.in‚20178.98046875 disk.rbd1.in‚ 2.01953125|h¿“Ô,‚(:ˆ ¶žìÒþ龕 gpu.0.gpu‚39.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û ÎÐ÷Õ,‚(:ì ¶žìÒ¡’Ò—! proc.memory.rssMB‚ 2809.890625 network.sent‚ 2.2703464e+07) disk./.usagePercent‚39.28873821013622 disk.dm-1.in‚ 0.1328125 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0# disk./.usageGB‚585.430591583252 disk.sda2.out‚0 disk.sda3.out‚ 366.56640625 disk.sda1.in‚0 disk.dm-1.out‚ 366.56640625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚30360.2587890625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚20178.98046875 cpu‚26.175501807431836" disk.dm-0.out‚30360.2587890625% disk.nvme0n1.out‚30360.2587890625 disk.rbd1.out‚ 40.6416015625 disk.rbd0.out‚0$ memory_percent‚97.50502109527588* proc.memory.percent‚17.150211334228516 network.recv‚ 3.277163e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 366.56640625" disk.nvme0n1.in‚20178.98046875( proc.memory.availableMB‚ 408.77734375 proc.cpu.threads‚17 disk.dm-0.in‚20178.98046875 disk.sda3.in‚ 0.1328125îiX&‘Ö,‚(:† ÅžìÒÉÞ¸“ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÍ(aô×,‚(:é ÅžìÒº¨Ê– disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.270855e+07 disk.dm-0.in‚20178.98046875! disk.dm-0.out‚30462.150390625 disk.sda.in‚ 0.1328125 disk.dm-1.out‚ 366.84765625 disk.drbd2165.out‚0 cpu‚26.17523081750868$ disk./.usageGB‚585.4296379089355 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 40.7041015625 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚20178.98046875! proc.memory.rssMB‚ 2809.890625 disk.sda.out‚ 366.84765625 disk.dm-1.in‚ 0.1328125 network.recv‚ 3.280931e+06 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 409.01953125 disk.sda3.out‚ 366.84765625 disk.rbd0.out‚0$ memory_percent‚97.50354290008545 disk.sda3.in‚ 0.1328125* proc.memory.percent‚17.150211334228516" disk.nvme0n1.in‚20178.98046875$ disk.nvme0n1.out‚30462.150390625 disk.sda1.out‚0& disk.nvme0c0n1.out‚30462.150390625* disk./.usagePercent‚39.288674208252665ì–ê4Ø,/ _wandbruntime‚12246  _runtime‚12246tµ¢ÉÙ,à & lr-AdamW/pg2‚8.883049397283825e-05 trainer/global_step‚14249 $ _timestamp‚1.784352587496771e+09 _runtime‚12246.192522094 _step‚1171 & lr-AdamW/pg1‚8.883049397283825e-05ˆ©àÎÚ,È trainer/global_step‚14249 $ _timestamp‚1.784352587496771e+09 _runtime‚12246.192522094 _step‚1171 & lr-AdamW/pg1‚8.883049397283825e-05 & lr-AdamW/pg2‚8.883049397283825e-05“ ïþ¤o4Û,/ _wandbruntime‚12246  _runtime‚12246¸º¶¦(Ü,P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131976945 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15230619807744 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112983170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2180824800187904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244960115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477040711395328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2165755346001920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15046820342347 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15046479792457 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180824800187904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131986922 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492271426071552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843525874997597e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15069812564480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15069812564480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970094 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15230714676224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24960328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30622395 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2165755346001920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477040692783104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15069471225344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2169662390827595 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 trainer/global_step‚14249 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2165755328962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15069471225344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112983172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2154615570485248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2180825158566400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112983170 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55582723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180825158566400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112983172 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55582991 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1172 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131976945 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2169662033238345 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24960331 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736ö‰à\ (Ý,†P H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131976945 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15230619807744 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112983170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2180824800187904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244960115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477040711395328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2165755346001920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15046820342347 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15046479792457 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180824800187904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131986922 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492271426071552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843525874997597e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15069812564480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15069812564480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970094 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15230714676224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24960328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30622395 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2165755346001920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477040692783104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15069471225344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2169662390827595 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 trainer/global_step‚14249 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2165755328962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15069471225344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112983172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2154615570485248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2180825158566400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112983170 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55582723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180825158566400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112983172 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55582991 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1172 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131976945 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2169662033238345 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24960331 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438” ²Ïƒ;4Þ,/ _wandbruntime‚12246  _runtime‚12246L—¢_‰ ß,‹N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2169809912553301 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112990804 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55586505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2169809554963939 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15070825198592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15070483852288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131985997 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15070825198592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30624694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986792 SEDeviceStatsMonitor.on_tra¶òˆin_batch_end/allocation.large_pool.allocated‚ 112990804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477074140153856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180973412230144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15231634801152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180973412230144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15070483852288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492305867719168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30624432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492305756342784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2165902587031552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2180973053844480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2154762081157120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2165902569992192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24962076 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112990802 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244976799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131985997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872Ÿ6-'à,N XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2169809912553301 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112990804 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55586505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2169809554963939 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15070825198592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15070483852288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131985997 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15070825198592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30624694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112990804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477074140153856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180973412230144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15231634801152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180973412230144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15070483852288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492305867719168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30624432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492305756342784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2165902587031552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2180973053844480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2154762081157120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2165902569992192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24962076 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112990802 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244976799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131985997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0• Xžöù4á,/ _wandbruntime‚12246  _runtime‚12246u)´Š â,… /loss/FRt1/detection_loss‚0.5130717681632422 0loss/SRqqt2/detection_loss‚0.542419225689222 % _timestamp‚1.7843525881231482e+09 0loss/FRt1/assignment_loss‚1.5108673412349263 0loss/FRt2/assignment_loss‚0.6588437272475498 2loss/FBt1/assignment_loss‚0.001563798223012066 2loss/FBt2/assignment_loss‚0.003065322638787095 epoch‚35 _runtime‚12246.816477789 2loss/SRqqt1/assignment_loss‚0.5063473150457138 .loss/FRt2/detection_loss‚1.496634508706073 /loss/FBt1/detection_loss‚0.4165723256993334 .loss/FBt2/detection_loss‚0.491043810854091 %loss/total_loss‚7.337314769786931 trainer/global_step‚14249 _step‚1174 3loss/SRqqt2/assignment_loss‚0.43581219660976683 1loss/SRqqt1/detection_loss‚0.76107342967521342¤Çrã,Š 2loss/FBt1/assignment_loss‚0.001563798223012066 2loss/FBt2/assignment_loss‚0.003065322638787095 epoch‚35 _runtime‚12246.816477789 2loss/SRqqt1/assignment_loss‚0.5063473150457138 .loss/FRt2/detection_loss‚1.496634508706073 /loss/FBt1/detection_loss‚0.4165723256993334 .loss/FBt2/detection_loss‚0.491043810854091 %loss/total_loss‚7.337314769786931 trainer/global_step‚14249 _step‚1174 3loss/SRqqt2/assignment_loss‚0.43581219660976683 1loss/SRqqt1/detection_loss‚0.7610734296752134 /loss/FRt1/detection_loss‚0.5130717681632422 0loss/SRqqt2/detection_loss‚0.542419225689222 % _timestamp‚1.7843525881231482e+09 0loss/FRt1/assignment_loss‚1.5108673412349263 0loss/FRt2/assignment_loss‚0.6588437272475498– @ˆ“ä,‚(:ˆ ÔžìÒÌöÞ” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚256.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Tzõå,‚(:ê ÔžìÒÍãÑ– disk.sda1.in‚0 cpu‚26.17499618548272 proc.cpu.threads‚17$ disk./.usageGB‚585.4295883178711 disk.dm-1.out‚ 367.1953125$ disk.nvme0c0n1.in‚20178.98046875 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.sda3.out‚ 367.1953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚30468.9873046875 disk.sda2.in‚0 network.recv‚ 3.284377e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 367.1953125* proc.memory.percent‚17.150211334228516 disk.rbd1.in‚ 2.01953125! proc.memory.rssMB‚ 2809.890625 disk.rbd1.out‚ 40.7041015625 disk.drbd2165.in‚0* disk./.usagePercent‚39.288670880154726" disk.nvme0n1.in‚20178.98046875 disk.sda3.in‚ 0.1328125 disk.dm-0.in‚20178.98046875" disk.dm-0.out‚30468.9873046875$ memory_percent‚97.50487804412842( proc.memory.availableMB‚ 408.80078125 network.sent‚ 2.2711514e+07% disk.nvme0n1.out‚30468.9873046875 disk.rbd0.out‚0 ,ý‘æ,‚(:† ãžìÒ´Îã“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s®7ˆäç,‚(:Ù ãžìÒÛ’‡– network.sent‚ 2.277482e+07$ disk./.usageGB‚585.3303833007812' disk.nvme0c0n1.out‚30477.1748046875" disk.dm-0.out‚30477.1748046875 disk.dm-1.out‚369.25 network.recv‚ 3.289192e+06" disk.nvme0n1.in‚20286.30078125% disk.nvme0n1.out‚30477.1748046875$ disk.nvme0c0n1.in‚20286.30078125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚17.150211334228516 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚369.25 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.out‚369.25 disk.sda3.in‚ 0.1328125! proc.memory.rssMB‚ 2809.890625 proc.cpu.threads‚17) disk./.usagePercent‚39.28201314821927 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0# memory_percent‚97.5050687789917 disk.dm-0.in‚20286.30078125 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 40.7666015625( proc.memory.availableMB‚ 408.76953125 disk.drbd2165.out‚0 cpu‚26.174744655740145ñ.{;žè,‚(:“ òžìÒ»ª• gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚240.541' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.108200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 o§„èé,‚(:Ý òžìÒé½Ý• disk.dm-0.in‚20286.30078125 disk.sda.in‚ 0.1328125 disk.drbd2165.in‚0 proc.cpu.threads‚17# disk./.usageGB‚585.377197265625 disk.rbd0.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.150211334228516 cpu‚26.174431767394154 network.sent‚ 2.2779917e+07 disk.rbd1.in‚ 2.01953125& disk.nvme0c0n1.out‚30489.708984375 disk.sda.out‚ 370.03125# memory_percent‚97.5031852722168& proc.memory.availableMB‚ 409.078125$ disk.nvme0n1.out‚30489.708984375 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 40.7666015625 disk.rbd0.out‚0 network.recv‚ 3.292372e+06) disk./.usagePercent‚39.28515487267947 disk.sda3.out‚ 370.03125 disk.dm-1.in‚ 0.1328125! disk.dm-0.out‚30489.708984375 disk.sda2.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2809.890625 disk.sda2.out‚0 disk.sda1.out‚0" disk.nvme0n1.in‚20286.30078125 disk.dm-1.out‚ 370.03125$ disk.nvme0c0n1.in‚20286.30078125uF :4ê,/ _wandbruntime‚12287  _runtime‚12287P'‘*Êë,Ä & lr-AdamW/pg1‚8.740121230710617e-05 & lr-AdamW/pg2‚8.740121230710617e-05 trainer/global_step‚14299 % _timestamp‚1.7843526285437596e+09 _runtime‚12287.240432809 _step‚1175VÑ-&Ïì,É _step‚1175 & lr-AdamW/pg1‚8.740121230710617e-05 & lr-AdamW/pg2‚8.740121230710617e-05 trainer/global_step‚14299 % _timestamp‚1.7843526285437596e+09 _runtime‚12287.240432809— O–'!4í,/ _wandbruntime‚12287  _runtime‚12287¢±5m%î,P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113364770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113364772 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 245806146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25047578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245796167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493993904690176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚14299 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55773410 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2173117380444160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478712130709504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2188237905549824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2173117380444160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478712149321728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2161941087039488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2173117397483520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15120508066304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15120508066304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15097436829255 _runtime‚12287.241377844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188237547171328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15281773980672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132431397 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173117397483520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493994020268032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15120166727168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132431397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113364772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 35758•Zãyè9250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113364770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843526285477269e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169?G¬5 (ï,†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245796167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493993904690176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚14299 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55773410 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2173117380444160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478712130709504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2188237905549824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2173117380444160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478712149321728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2161941087039488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2173117397483520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15120508066304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15120508066304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15097436829255 _runtime‚12287.241377844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188237547171328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15281773980672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132431397 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173117397483520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493994020268032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15120166727168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132431397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113364772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113364770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843526285477269e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113364770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113364772 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 245806146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25047578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888˜ 4%¶‡4ð,/ _wandbruntime‚12287  _runtime‚12287|úa¥'ñ,ŠN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113372404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚494028353070592 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2177186062633809 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15098107333087 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132440449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55777143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15282793602560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2188386159213568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245812851 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478745578080256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55776872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30727817 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2173264621473792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30727549 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15121179354112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15121520700416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132450440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2188386159213568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚494028466544128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843526291653001e+09ܺ 'ò,N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245812851 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478745578080256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55776872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30727817 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2173264621473792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30727549 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15121179354112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15121520700416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132450440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2188386159213568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚494028466544128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843526291653001e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113372404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚494028353070592 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2177186062633809 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15098107333087 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132440449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55777143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15282793602560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2188386159213568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591™ À7„4ó,/ _wandbruntime‚12287  _runtime‚12287ÿ$aô,Š epoch‚35 _runtime‚12287.862028252 _step‚1178 3loss/FBt1/assignment_loss‚7.545939250472125e-05 /loss/FRt1/detection_loss‚0.4642949747914792 1loss/SRqqt1/detection_loss‚0.8023804808439245 trainer/global_step‚14299 % _timestamp‚1.7843526291687176e+09 3loss/SRqqt2/assignment_loss‚0.41652205170527007 /loss/FBt2/detection_loss‚0.4170038045979967 %loss/total_loss‚7.084296881933879 0loss/FRt1/assignment_loss‚1.4978155252479373 2loss/SRqqt1/assignment_loss‚0.4309926650163231 0loss/FRt2/assignment_loss‚0.5478854285622085 3loss/FBt2/assignment_loss‚0.0001897879423110643 /loss/FRt2/detection_loss‚1.3215243366710099 1loss/SRqqt2/detection_loss‚0.5568120657946436 /loss/FBt1/detection_loss‚0.6288003013682707”«õ, 0loss/FRt2/assignment_loss‚0.5478854285622085 3loss/FBt2/assignment_loss‚0.0001897879423110643 /loss/FRt2/detection_loss‚1.3215243366710099 1loss/SRqqt2/detection_loss‚0.5568120657946436 /loss/FBt1/detection_loss‚0.6288003013682707 epoch‚35 _runtime‚12287.862028252 _step‚1178 3loss/FBt1/assignment_loss‚7.545939250472125e-05 /loss/FRt1/detection_loss‚0.4642949747914792 1loss/SRqqt1/detection_loss‚0.8023804808439245 trainer/global_step‚14299 % _timestamp‚1.7843526291687176e+09 3loss/SRqqt2/assignment_loss‚0.41652205170527007 /loss/FBt2/detection_loss‚0.4170038045979967 %loss/total_loss‚7.084296881933879 0loss/FRt1/assignment_loss‚1.4978155252479373 2loss/SRqqt1/assignment_loss‚0.4309926650163231š »æZ÷ö,‚(:ˆ ŸìÒ„™¾“©ç•x gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚242.148' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.4296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹šù÷,‚(:î ŸìÒÉ“ó–& disk.nvme0n1.out‚30503.06787109375 disk.sda3.in‚ 0.1328125$ disk./.usageGB‚585.3853988647461 disk.rbd0.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.150211334228516 network.recv‚ 3.296111e+06 disk.dm-0.in‚20286.30078125 disk.rbd1.out‚ 40.8291015625 network.sent‚ 2.2784918e+07$ disk.nvme0c0n1.in‚20286.30078125 disk.sda.out‚ 372.23046875 disk.rbd1.in‚ 2.01953125% proc.memory.availableMB‚ 409.03125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚30503.06787109375! proc.memory.rssMB‚ 2809.890625 disk.sda1.in‚0 disk.dm-1.out‚ 372.23046875 disk.sda3.out‚ 372.23046875 disk.sda1.out‚0( disk.nvme0c0n1.out‚30503.06787109375 disk.sda2.out‚0$ memory_percent‚97.50347137451172 cpu‚26.174275157118412 proc.cpu.threads‚17* disk./.usagePercent‚39.285705288878084 disk.dm-1.in‚ 0.1328125" disk.nvme0n1.in‚20286.30078125N“ø,‚(:ˆ ŸìÒ‘¡Ë“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.9418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ow÷ù,‚(:ì ŸìÒâÏ–) disk./.usagePercent‚39.28971539089442 disk.drbd2165.in‚0& disk.nvme0n1.out‚30512.32958984375 disk.sda1.out‚0" disk.nvme0n1.in‚20286.30078125 disk.rbd0.in‚0 disk.sda.out‚ 372.71484375$ disk./.usageGB‚585.4451522827148 disk.sda2.in‚0* proc.memory.percent‚17.150211334228516 network.recv‚ 3.300223e+06 disk.dm-0.in‚20286.30078125 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 40.8291015625 cpu‚26.174128064712658 disk.dm-1.in‚ 0.1328125# memory_percent‚97.5031852722168 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 network.sent‚ 2.284821e+07 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚30512.32958984375 disk.sda2.out‚0# disk.dm-0.out‚30512.32958984375 disk.sda3.out‚ 372.71484375 disk.rbd0.out‚0& proc.memory.availableMB‚ 409.078125 disk.dm-1.out‚ 372.71484375$ disk.nvme0c0n1.in‚20286.30078125! proc.memory.rssMB‚ 2809.890625 proc.cpu.threads‚17gÀµØ4ú,/ _wandbruntime‚12327  _runtime‚12327°)¾Çû,Á _runtime‚12327.98541494 _step‚1179 & lr-AdamW/pg1‚8.598073871125939e-05 & lr-AdamW/pg2‚8.598073871125939e-05 trainer/global_step‚14349 # _timestamp‚1.78435266929179e+09²Œ[‹Ìü,Æ trainer/global_step‚14349 # _timestamp‚1.78435266929179e+09 _runtime‚12327.98541494 _step‚1179 & lr-AdamW/pg1‚8.598073871125939e-05 & lr-AdamW/pg2‚8.598073871125939e-05› ]žÃ4ý,/ _wandbruntime‚12327  _runtime‚12327‡„(þ,€P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2180479448965120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495716548829696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚12327.98668551 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30828947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2195650652533248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15332961581568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246642198 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25134828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113746372 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132885849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132885849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495716443737600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132895826 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2195650294154752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55963775 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚480383568635904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2184414333399361 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132895826 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2195650294154752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2184414690988611 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113746372 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2195650652533248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25134831 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15148053316163 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55964043 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚14349Å{ýÿ (ÿ,…P GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2195650294154752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2184414690988611 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113746372 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2195650652533248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25134831 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15148053316163 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55964043 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚14349 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2180479448965120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495716548829696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚12327.98668551 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30828947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2195650652533248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15332961581568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246642198 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25134828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113746372 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132885849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132885849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495716443737600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132895826 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2195650294154752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55963775 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚480383568635904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2184414333399361 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132895826 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0œ r,¬A4€-/ _wandbruntime‚12328  _runtime‚12328@ýÍq"-‹N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2195798547811328 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55967605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15149064369997 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2195798906196992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113754004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495750890487808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30831296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15172216202240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55967872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15148723819995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113754002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2180626689994752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15172216202240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚480416997394432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2169413148344320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2180626689994752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚14349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25136576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚12328.600136621 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246648903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15333893093376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113754002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132904892 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2180626672955392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30831032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843526699063933e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2180626672955392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2195798906196992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648903 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184562212714317 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15171874855936 VHDeviceStatsMonitor.on_train_batch_end/requested_by ´Ÿc tes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132894901 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2169413131304960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132904892 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495750997669888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚480417016006656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976våV'‚-N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132904892 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2180626672955392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30831032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843526699063933e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2180626672955392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2195798906196992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648903 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184562212714317 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15171874855936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132894901 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2169413131304960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132904892 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495750997669888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚480417016006656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2195798547811328 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55967605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15149064369997 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2195798906196992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113754004 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495750890487808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30831296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15172216202240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55967872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15148723819995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113754002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2180626689994752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15172216202240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚480416997394432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2169413148344320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2180626689994752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚14349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25136576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚12328.600136621 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246648903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15333893093376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113754002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 †>žÄ4ƒ-/ _wandbruntime‚12328  _runtime‚12328éÉÿµ „-‡ 0loss/FRt1/assignment_loss‚1.4451119924465279 2loss/FBt1/assignment_loss‚0.002038754675671347 1loss/SRqqt1/detection_loss‚0.7882210611435657 /loss/SRqqt2/detection_loss‚0.58193571920936 _runtime‚12328.603118623 .loss/FRt2/detection_loss‚1.336755240009008 epoch‚35 %loss/total_loss‚6.993606035627016 0loss/FRt2/assignment_loss‚0.5032980922422453 /loss/FRt1/detection_loss‚0.5328732920567456 /loss/FBt1/detection_loss‚0.3901493845623918 0loss/FBt2/detection_loss‚0.45161572174771764 % _timestamp‚1.7843526699098177e+09 _step‚1182 2loss/SRqqt1/assignment_loss‚0.4934716512988476 3loss/SRqqt2/assignment_loss‚0.46571366360907235 3loss/FBt2/assignment_loss‚0.0024214626258618863 trainer/global_step‚143498a‰ …-Œ /loss/FRt1/detection_loss‚0.5328732920567456 /loss/FBt1/detection_loss‚0.3901493845623918 0loss/FBt2/detection_loss‚0.45161572174771764 % _timestamp‚1.7843526699098177e+09 _step‚1182 2loss/SRqqt1/assignment_loss‚0.4934716512988476 3loss/SRqqt2/assignment_loss‚0.46571366360907235 3loss/FBt2/assignment_loss‚0.0024214626258618863 trainer/global_step‚14349 0loss/FRt1/assignment_loss‚1.4451119924465279 2loss/FBt1/assignment_loss‚0.002038754675671347 1loss/SRqqt1/detection_loss‚0.7882210611435657 /loss/SRqqt2/detection_loss‚0.58193571920936 _runtime‚12328.603118623 .loss/FRt2/detection_loss‚1.336755240009008 epoch‚35 %loss/total_loss‚6.993606035627016 0loss/FRt2/assignment_loss‚0.5032980922422453ž æÕ,F“†-‚(:ˆ ŸŸìÒš¥°“ gpu.0.gpu‚77.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x3Glð‡-‚(:å ŸŸìÒïÄÒ•" disk.nvme0n1.in‚20357.12109375 disk.dm-1.out‚ 373.53125 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.dm-0.in‚20357.12109375# disk./.usageGB‚585.443775177002 cpu‚26.17428246323351& disk.nvme0n1.out‚30517.28564453125) disk./.usagePercent‚39.28962297217456 disk.sda3.out‚ 373.53125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.sda.out‚ 373.53125 disk.sda2.out‚0 network.sent‚ 2.2853091e+07 disk.rbd1.out‚ 40.8916015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚20357.12109375# disk.dm-0.out‚30517.28564453125 disk.sda.in‚ 0.1328125$ memory_percent‚97.50692844390869 disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 408.46484375 network.recv‚ 3.303791e+06( disk.nvme0c0n1.out‚30517.28564453125! proc.memory.rssMB‚ 2809.890625* proc.memory.percent‚17.150211334228516¥Ÿ§z“ˆ-‚(:ˆ ®ŸìÒÝ𤓠gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙªŠ×ù‰-‚(:î ®ŸìÒڢė network.recv‚ 3.308363e+06 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0* proc.memory.percent‚17.150211334228516 disk.sda1.out‚0# disk.dm-0.out‚30535.18603515625 disk.sda.in‚ 0.1328125 network.sent‚ 2.2916351e+07$ disk./.usageGB‚585.5345916748047$ disk.nvme0c0n1.in‚20357.12109375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚20357.12109375 disk.sda.out‚ 375.41796875 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.29571774354222 disk.dm-1.out‚ 375.41796875 disk.drbd2165.in‚0& disk.nvme0n1.out‚30535.18603515625 disk.sda3.out‚ 375.41796875( proc.memory.availableMB‚ 408.76953125 disk.rbd1.out‚ 40.8916015625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚30535.18603515625" disk.nvme0n1.in‚20357.12109375 disk.sda2.in‚0# memory_percent‚97.5050687789917 cpu‚26.17460838848364! proc.memory.rssMB‚ 2809.890625ªÈ6ð“Š-‚(:ˆ ½ŸìÒþÀÙ• gpu.0.gpu‚86.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¢ª)ì‹-‚(:á ½ŸìÒêÁ— network.sent‚ 2.2923401e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17 disk.sda3.out‚ 376.3203125 cpu‚26.174780480982793 disk.sda.in‚ 0.1328125 disk.sda.out‚ 376.3203125 disk.sda2.out‚0 disk.rbd1.out‚ 40.9541015625 disk.drbd2165.out‚0$ disk./.usageGB‚585.4151878356934 disk.drbd2165.in‚0 disk.dm-0.in‚20357.12109375 disk.rbd0.in‚0 disk.dm-1.out‚ 376.3203125$ memory_percent‚97.50652313232422 network.recv‚ 3.311844e+06 disk.rbd0.out‚0) disk./.usagePercent‚39.28770445171296" disk.nvme0n1.in‚20357.12109375% proc.memory.availableMB‚ 408.53125! proc.memory.rssMB‚ 2809.890625$ disk.nvme0c0n1.in‚20357.12109375% disk.nvme0c0n1.out‚30659.20703125 disk.dm-1.in‚ 0.1328125 disk.dm-0.out‚30659.20703125* proc.memory.percent‚17.150211334228516 disk.sda2.in‚0# disk.nvme0n1.out‚30659.20703125 disk.sda3.in‚ 0.1328125e;:4Œ-/ _wandbruntime‚12368  _runtime‚12368tMT©Ê-Ä & lr-AdamW/pg1‚8.456916441966237e-05 & lr-AdamW/pg2‚8.456916441966237e-05 trainer/global_step‚14399 % _timestamp‚1.7843527098357654e+09 _runtime‚12368.529325158 _step‚1183žØ:ÏŽ-É & lr-AdamW/pg2‚8.456916441966237e-05 trainer/global_step‚14399 % _timestamp‚1.7843527098357654e+09 _runtime‚12368.529325158 _step‚1183 & lr-AdamW/pg1‚8.456916441966237e-05Ÿ ‘)…4-/ _wandbruntime‚12368  _runtime‚12368JM[^(-€P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2203063041138176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114127972 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15198329253181 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25222078 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497439154826240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2191790841069119 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25222081 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2176592171266048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784352709836833e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114127970 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15384129651712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2191790483479869 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133350278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497439049734144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2187841500446720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482055006562304 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482055025174528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚14399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30932416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2203063041138176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15198669803071 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15221557730816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114127970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30932139 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2203063041138176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15198669803071 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15221557730816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114127970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30932139 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25222078 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497439154826240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2191790841069119 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25222081 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2176592171266048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784352709836833e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114127970 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15384129651712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2191790483479869 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133350278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497439049734144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2187841500446720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482055006562304 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482055025174528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169  â)DË4’-/ _wandbruntime‚12369  _runtime‚12369×U¾«'“-‹N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114135604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25223826 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚497473495204352 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114135602 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133359344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15222911704064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚12369.145035127 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133349353 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2203211294794752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15385059883520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15222911704064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497473612872192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2191938362794825 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2176738664898560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247494948 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133359344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114135602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25223823 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30934193 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15385158939136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15199340306903 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15222570357760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2191938005205463 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15199680856905 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56158016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30934460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2176738681937920 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚14399 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114135604 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15222570357760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2191938005205463 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15199680856905 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56158016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30934460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2176738681937920 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚14399 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114135604 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133349353 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2203211294794752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15385059883520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15222911704064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497473612872192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2191938362794825 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2176738664898560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247494948 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133359344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114135602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25223823¡ ØùoÐ4•-/ _wandbruntime‚12369  _runtime‚12369›_Uá–-Š 3loss/SRqqt2/assignment_loss‚0.43272132303277616 /loss/FRt2/detection_loss‚1.3998482492422897 1loss/SRqqt2/detection_loss‚0.5887700889303342 0loss/FBt2/detection_loss‚0.37781338036540446 epoch‚35 /loss/FRt1/detection_loss‚0.5199993637454623 0loss/FBt1/detection_loss‚0.37523067001186844 %loss/total_loss‚7.205297611351725 _step‚1186 /loss/FRt1/assignment_loss‚1.557706745383545 0loss/FRt2/assignment_loss‚0.6305631150333386 2loss/SRqqt1/assignment_loss‚0.4536063671536946 3loss/FBt1/assignment_loss‚8.519604737265192e-05 3loss/FBt2/assignment_loss‚0.0017574415712067629 0loss/SRqqt1/detection_loss‚0.867195670834432 trainer/global_step‚14399 % _timestamp‚1.7843527104545674e+09 _runtime‚12369.147894063Ø£¸—- /loss/FRt1/assignment_loss‚1.557706745383545 0loss/FRt2/assignment_loss‚0.6305631150333386 2loss/SRqqt1/assignment_loss‚0.4536063671536946 3loss/FBt1/assignment_loss‚8.519604737265192e-05 3loss/FBt2/assignment_loss‚0.0017574415712067629 0loss/SRqqt1/detection_loss‚0.867195670834432 trainer/global_step‚14399 % _timestamp‚1.7843527104545674e+09 _runtime‚12369.147894063 3loss/SRqqt2/assignment_loss‚0.43272132303277616 /loss/FRt2/detection_loss‚1.3998482492422897 1loss/SRqqt2/detection_loss‚0.5887700889303342 0loss/FBt2/detection_loss‚0.37781338036540446 epoch‚35 /loss/FRt1/detection_loss‚0.5199993637454623 0loss/FBt1/detection_loss‚0.37523067001186844 %loss/total_loss‚7.205297611351725 _step‚1186¢ /ä̘-‚(:’ ÌŸìÒ Ïí“ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÒG(ø™-‚(:í ÌŸìÒ¡Šö• disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125" disk.dm-0.out‚30678.2138671875 disk.sda.out‚ 380.41796875 disk.sda2.out‚0* proc.memory.percent‚17.150211334228516 cpu‚26.176631927791032 disk.rbd0.in‚0 disk.dm-0.in‚20357.12109375 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ disk./.usageGB‚585.4038047790527 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚30678.2138671875" disk.nvme0n1.in‚20357.12109375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 380.41796875 network.sent‚ 2.2925406e+07 disk.dm-1.out‚ 380.41796875 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.28694052523079% disk.nvme0n1.out‚30678.2138671875( proc.memory.availableMB‚ 408.80078125 disk.rbd1.out‚ 40.9541015625$ memory_percent‚97.50487804412842 network.recv‚ 3.314356e+06$ disk.nvme0c0n1.in‚20357.12109375 disk.sda.in‚ 0.1328125! proc.memory.rssMB‚ 2809.890625Ži¥4š-/ _wandbruntime‚12386  _runtime‚12386:)ù-›-ž{ 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8514360785484314 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6426616907119751 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24167156219482422 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884835481643677 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961845874786377 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3703838586807251 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9965986609458923 ,particle/accuracy_4_of_4‚0.9332235455513 *loss/val_total_loss‚7.6357437812009135 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406724572181702 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8372977375984192 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6validation_average_jet_accuracy‚0.3901576101779938 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.679172694683075 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12895135581493378 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11425670981407166 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6257197260856628 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7270877957344055 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/sensitivity‚0.6004812717437744 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30430206656455994 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5492714643478394 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022307876497507095 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7935711145401001 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22895094752311707 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32289260625839233 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2163102775812149 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35691606998443604 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 epoch‚35 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7842906713485718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 *jet/accuracy_1_of_3‚0.9202289581298828 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7911429405212402 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.87409454¢™ÀŽ-5841217 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7182213664054871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7254263758659363 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021109197288751602 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.03999513015151024 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.817630410194397 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31887438893318176 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3556586802005768 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8296673893928528 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚14399 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8367913365364075 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10388374328613281 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8786501288414001 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3226393759250641 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7907546758651733 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7002140879631042 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6664633750915527 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23429857194423676 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3329160213470459 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25166216492652893 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.586218535900116 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3132025897502899 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6332953572273254 'particle/f_score‚0.6131594777107239 jet/accuracy_4_of_5‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_4_of_4‚0.01670061983168125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647242546081543 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6612566113471985 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018042031675577164 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6362466812133789 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6236268281936646 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2545040249824524 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5334367156028748 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036862898617982864 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2439214438199997 *jet/accuracy_2_of_4‚0.7642937898635864 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8367913365364075 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022644182667136192 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14599864184856415 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7640023231506348 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1311802864074707 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3133035898208618 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047083668410778046 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7909387946128845 particle/accuracy_2_of_2‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.778807520866394 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6704784631729126 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _runtime‚12386.894449647 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952049255371094 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05391387641429901 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9020605683326721 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16251890361309052 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.740048885345459 particle/accuracy_2_of_3‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7194980382919312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23498523235321045 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.567537784576416 +jet/accuracy_1_of_1‚0.36106640100479126 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957960247993469 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18343736231327057 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7857060432434082 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12391668558120728 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5583866834640503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6799880862236023 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3133035898208618 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23251651227474213 )jet/accuracy_3_of_4‚0.040353924036026 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6463865041732788 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7242432832717896 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31251558661460876 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6796020865440369 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7084169983863831 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03647688031196594 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5245833396911621 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 +jet/accuracy_2_of_2‚0.13106754422187805 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38189271092414856 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24020807445049286 *jet/accuracy_2_of_3‚0.5130007863044739 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7714765667915344 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6465624570846558 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.302406370639801 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4887564480304718 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2447989135980606 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7394086122512817 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48912855982780457 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02091936394572258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6591636538505554 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6465624570846558 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7488743662834167 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_1_of_5‚1 _step‚1187 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26504918932914734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05917489528656006 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950152635574341 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018042031675577164 *jet/accuracy_1_of_4‚0.9778740406036377 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1142873540520668 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843527282001393e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7772688269615173 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7164638042449951 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26264938712120056 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6985136866569519 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.874094545841217 ("Purity/2FRt1SRqqt0FBt/event_purity‚0a&OÄ©=œ-£{ :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35691606998443604 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 epoch‚35 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7842906713485718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 *jet/accuracy_1_of_3‚0.9202289581298828 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7911429405212402 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.874094545841217 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7182213664054871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7254263758659363 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021109197288751602 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.03999513015151024 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.817630410194397 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31887438893318176 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3556586802005768 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8296673893928528 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚14399 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8367913365364075 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10388374328613281 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8786501288414001 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3226393759250641 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7907546758651733 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7002140879631042 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6664633750915527 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23429857194423676 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3329160213470459 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25166216492652893 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.586218535900116 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3132025897502899 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6332953572273254 'particle/f_score‚0.6131594777107239 jet/accuracy_4_of_5‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_4_of_4‚0.01670061983168125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647242546081543 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6612566113471985 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018042031675577164 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6362466812133789 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6236268281936646 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2545040249824524 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5334367156028748 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036862898617982864 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2439214438199997 *jet/accuracy_2_of_4‚0.7642937898635864 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8367913365364075 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022644182667136192 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14599864184856415 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7640023231506348 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1311802864074707 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3133035898208618 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047083668410778046 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7909387946128845 particle/accuracy_2_of_2‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.778807520866394 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6704784631729126 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _runtime‚12386.894449647 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952049255371094 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05391387641429901 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9020605683326721 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16251890361309052 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.740048885345459 particle/accuracy_2_of_3‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7194980382919312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23498523235321045 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.567537784576416 +jet/accuracy_1_of_1‚0.36106640100479126 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957960247993469 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18343736231327057 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7857060432434082 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12391668558120728 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5583866834640503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6799880862236023 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3133035898208618 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23251651227474213 )jet/accuracy_3_of_4‚0.040353924036026 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6463865041732788 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7242432832717896 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31251558661460876 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6796020865440369 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7084169983863831 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03647688031196594 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5245833396911621 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 +jet/accuracy_2_of_2‚0.13106754422187805 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38189271092414856 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24020807445049286 *jet/accuracy_2_of_3‚0.5130007863044739 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7714765667915344 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6465624570846558 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.302406370639801 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4887564480304718 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2447989135980606 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7394086122512817 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48912855982780457 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02091936394572258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6591636538505554 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6465624570846558 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7488743662834167 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_1_of_5‚1 _step‚1187 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26504918932914734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05917489528656006 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950152635574341 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018042031675577164 *jet/accuracy_1_of_4‚0.9778740406036377 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1142873540520668 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843527282001393e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7772688269615173 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7164638042449951 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26264938712120056 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6985136866569519 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.874094545841217 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8514360785484314 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6426616907119751 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24167156219482422 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884835481643677 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961845874786377 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3703838586807251 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9965986609458923 ,particle/accuracy_4_of_4‚0.9332235455513 *loss/val_total_loss‚7.6357437812009135 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406724572181702 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8372977375984192 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6validation_average_jet_accuracy‚0.3901576101779938 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.679172694683075 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12895135581493378 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11425670981407166 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6257197260856628 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7270877957344055 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/sensitivity‚0.6004812717437744 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30430206656455994 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5492714643478394 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022307876497507095 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7935711145401001 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22895094752311707 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.32289260625839233 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2163102775812149 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875£ Ñ4›e“-‚(:ˆ ÛŸìÒñƒ• gpu.0.gpu‚66.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚234.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0єȰëž-‚(:à ÛŸìÒõ‘¹–$ disk./.usageGB‚585.3915557861328 disk.sda.in‚ 0.1328125 network.sent‚ 2.2988563e+07 disk.drbd2165.out‚0 disk.dm-0.in‚ 20446.1875 disk.sda2.in‚0 disk.rbd1.out‚ 41.0166015625 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 401.0546875& disk.nvme0n1.out‚30812.52587890625 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 383.8828125 disk.rbd0.in‚0! proc.memory.rssMB‚ 2817.796875* proc.memory.percent‚17.198467254638672$ memory_percent‚97.55215644836426 proc.cpu.threads‚17 network.recv‚ 3.319195e+06 disk.dm-1.out‚ 383.8828125# disk.dm-0.out‚30812.52587890625 disk.sda.out‚ 383.8828125 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚30812.52587890625 disk.nvme0n1.in‚ 20446.1875* disk./.usagePercent‚39.286118485038344 disk.dm-1.in‚ 0.1328125 disk.nvme0c0n1.in‚ 20446.1875 disk.rbd0.out‚0 cpu‚26.18005994079217‚O«Ü“Ÿ-‚(:ˆ êŸìÒª—†“ gpu.0.gpu‚53.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚216.303' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.2606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:/žâ -‚(:× êŸìҪブ cpu‚26.17950282040956( proc.memory.availableMB‚ 401.08203125 network.recv‚ 3.323692e+06 disk.sda1.out‚0 disk.dm-1.out‚ 386.1484375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 20446.1875# disk.nvme0n1.out‚30875.06640625$ disk./.usageGB‚585.4388160705566 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125$ memory_percent‚97.55198955535889* proc.memory.percent‚17.198467254638672 network.sent‚ 2.3046088e+07 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 20446.1875 disk.dm-0.out‚30875.06640625 disk.sda.in‚ 0.1328125) disk./.usagePercent‚39.28929016238005 disk.sda.out‚ 386.1484375 disk.nvme0n1.in‚ 20446.1875 disk.sda3.out‚ 386.1484375 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚30875.06640625 disk.rbd1.in‚ 2.01953125! proc.memory.rssMB‚ 2817.796875 disk.rbd1.out‚ 41.0205078125AÒ “¡-‚(:ˆ ùŸìÒ«ÊΓ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jä/¾Ü¢-‚(:Ñ ùŸìÒÂ㳕 disk.nvme0n1.in‚ 20446.1875 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0) disk./.usagePercent‚39.29206886815669 disk.dm-0.in‚ 20446.1875 disk.drbd2165.in‚0 cpu‚26.178947817331906$ disk.nvme0c0n1.out‚ 30878.2421875$ disk./.usageGB‚585.4802207946777 disk.sda3.out‚ 386.5546875 disk.nvme0c0n1.in‚ 20446.1875 proc.cpu.threads‚17 network.sent‚ 2.305107e+07" disk.nvme0n1.out‚ 30878.2421875 disk.rbd1.out‚ 41.0830078125 disk.rbd0.in‚0 disk.drbd2165.out‚0# memory_percent‚97.5520133972168& proc.memory.availableMB‚ 401.078125 disk.dm-1.out‚ 386.5546875* proc.memory.percent‚17.198467254638672 network.recv‚ 3.326964e+06 disk.sda.out‚ 386.5546875! proc.memory.rssMB‚ 2817.796875 disk.dm-0.out‚ 30878.2421875 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.1328125ì’÷¨4£-/ _wandbruntime‚12426  _runtime‚12426×:='ʤ-Ä & lr-AdamW/pg2‚8.316658009509405e-05 trainer/global_step‚14449 % _timestamp‚1.7843527676467385e+09 _runtime‚12426.340322228 _step‚1188 & lr-AdamW/pg1‚8.316658009509405e-05Ãl}QÏ¥-É & lr-AdamW/pg1‚8.316658009509405e-05 & lr-AdamW/pg2‚8.316658009509405e-05 trainer/global_step‚14449 % _timestamp‚1.7843527676467385e+09 _runtime‚12426.340322228 _step‚1188¤ ±£Ðü4¦-/ _wandbruntime‚12426  _runtime‚12426Ã·Ü §-P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2197133101539328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2212423446355968 E;DeviceStatsMonÊ0ãü%itor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483742356588544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2212423087977472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114630238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133865075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483742375200768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15266665465318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114630238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133875052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2201093016499686 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2185826368073728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843527676478398e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505290 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2197133118578688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31068243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15290327777280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499195710989824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133875052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25311851 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212423087977472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15290327777280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14449 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114630236 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15289986438144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15453438784000 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15267006015208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248495311 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2197133101539328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31068499 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56380350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2212423446355968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114630236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2185826351034368Ñ't (¨-†P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15290327777280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499195710989824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133875052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25311851 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212423087977472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15290327777280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14449 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114630236 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15289986438144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15453438784000 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15267006015208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248495311 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2197133101539328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31068499 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56380350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2212423446355968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114630236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2185826351034368 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2197133101539328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2212423446355968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483742356588544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2212423087977472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114630238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133865075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483742375200768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15266665465318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114630238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133875052 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2201093016499686 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2185826368073728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843527676478398e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505290 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2197133118578688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31068243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088¥ Vð¹4©-/ _wandbruntime‚12426  _runtime‚12426T+Á'ª-‹N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2212571700019712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15267676519040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2197280342568960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133884118 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499230168332288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2185972878745600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2197280342568960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268017069042 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114637868 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133874127 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15454382985216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521988 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15290999065088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2197280359608320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56384181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31070332 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521988 _runtime‚12426.955773363 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483775803959296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483775785347072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56383925 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499230271320064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚14449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25313596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133874127 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2201240895814642 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843527682620208e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133884118 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2212571341634048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2185972861706240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888ä¾l™ «-N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15267676519040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2197280342568960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133884118 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499230168332288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2185972878745600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2197280342568960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268017069042 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114637868 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133874127 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚1545438298521Ëáú†6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521988 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15290999065088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2197280359608320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56384181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31070332 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521988 _runtime‚12426.955773363 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483775803959296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483775785347072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56383925 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499230271320064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚14449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25313596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133874127 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2201240895814642 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843527682620208e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133884118 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2212571341634048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2185972861706240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2212571700019712¦ ÁžT4¬-/ _wandbruntime‚12426  _runtime‚12426mŒ°J­-‰ 3loss/FBt2/assignment_loss‚0.0009298261592355301 /loss/FBt2/detection_loss‚0.4252618014301491 epoch‚36 _step‚1191 %loss/total_loss‚6.850987670214725 0loss/FRt1/assignment_loss‚1.4828757823992322 2loss/SRqqt1/assignment_loss‚0.4667998648222865 /loss/FRt1/detection_loss‚0.5472266890123874 /loss/FRt2/detection_loss‚1.3191105112010977 trainer/global_step‚14449 3loss/FBt1/assignment_loss‚0.0010660223961027762 1loss/SRqqt2/detection_loss‚0.5991103874502092 % _timestamp‚1.7843527682650936e+09 _runtime‚12426.95838863 0loss/FRt2/assignment_loss‚0.5270964786374768 2loss/SRqqt2/assignment_loss‚0.4275213090708518 1loss/SRqqt1/detection_loss‚0.8089312961765419 0loss/FBt1/detection_loss‚0.24505770145915448:*r®-Ž 0loss/FRt2/assignment_loss‚0.5270964786374768 2loss/SRqqt2/assignment_loss‚0.4275213090708518 1loss/SRqqt1/detection_loss‚0.8089312961765419 0loss/FBt1/detection_loss‚0.24505770145915448 3loss/FBt2/assignment_loss‚0.0009298261592355301 /loss/FBt2/detection_loss‚0.4252618014301491 epoch‚36 _step‚1191 %loss/total_loss‚6.850987670214725 0loss/FRt1/assignment_loss‚1.4828757823992322 2loss/SRqqt1/assignment_loss‚0.4667998648222865 /loss/FRt1/detection_loss‚0.5472266890123874 /loss/FRt2/detection_loss‚1.3191105112010977 trainer/global_step‚14449 3loss/FBt1/assignment_loss‚0.0010660223961027762 1loss/SRqqt2/detection_loss‚0.5991103874502092 % _timestamp‚1.7843527682650936e+09 _runtime‚12426.95838863§ ¾ÆBø“¯-‚(:ˆ ˆ ìÒÒ¥º“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òY§Òç°-‚(:Ü ˆ ìÒí¯Æ– network.recv‚ 3.330189e+06$ disk./.usageGB‚585.4531021118164 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 41.0830078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 20446.1875 disk.rbd0.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚30932.88525390625* proc.memory.percent‚17.198467254638672 disk.dm-1.out‚ 386.8359375# disk.dm-0.out‚30932.88525390625 cpu‚26.17886785545499 network.sent‚ 2.3056082e+07( disk.nvme0c0n1.out‚30932.88525390625 disk.sda.out‚ 386.8359375# memory_percent‚97.5534439086914% proc.memory.availableMB‚ 400.84375) disk./.usagePercent‚39.29024891059578 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 20446.1875! proc.memory.rssMB‚ 2817.796875 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 386.8359375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 20446.1875 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda.in‚ 0.1328125±ÃÉU“±-‚(:ˆ — ìÒêŽÓ” gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eîj¬ü²-‚(:ñ — ìÒ²­• disk.rbd0.out‚0$ disk.nvme0c0n1.in‚20535.80859375& disk.nvme0n1.out‚30984.07958984375 network.sent‚ 2.3119443e+07* disk./.usagePercent‚39.292836122736816$ disk./.usageGB‚585.4916534423828 disk.sda3.out‚ 387.04296875 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.out‚0* proc.memory.percent‚17.198467254638672 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 41.2236328125 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚30984.07958984375 cpu‚26.179138899972127 disk.sda1.out‚0 disk.dm-0.in‚20535.80859375# disk.dm-0.out‚30984.07958984375( proc.memory.availableMB‚ 401.05859375! proc.memory.rssMB‚ 2817.796875 disk.drbd2165.in‚0" disk.nvme0n1.in‚20535.80859375$ memory_percent‚97.55213260650635 network.recv‚ 3.335094e+06 disk.dm-1.out‚ 387.04296875 disk.sda.out‚ 387.04296875 disk.sda2.in‚0X°Ä“³-‚(:ˆ ¦ ìÒ™Š™“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§YÝæñ´-‚(:æ ¦ ì҆䨖 network.recv‚ 3.338589e+06% disk.nvme0n1.out‚30990.4990234375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚20535.80859375 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 387.7890625* proc.memory.percent‚17.198467254638672 cpu‚26.17935983874037 disk.sda.out‚ 387.7890625$ proc.memory.availableMB‚401.0625 network.sent‚ 2.3124306e+07 disk.rbd1.out‚ 41.2236328125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125! proc.memory.rssMB‚ 2817.796875* disk./.usagePercent‚39.289567674547165 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 387.7890625 disk.rbd1.in‚ 2.01953125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚20535.80859375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.55210876464844$ disk./.usageGB‚585.4429512023926 disk.sda2.out‚0" disk.nvme0n1.in‚20535.80859375' disk.nvme0c0n1.out‚30990.4990234375" disk.dm-0.out‚30990.4990234375ºæÛT4µ-/ _wandbruntime‚12466  _runtime‚12466ž >ȶ- _runtime‚12466.883388331 _step‚1192 % lr-AdamW/pg1‚8.17730758229248e-05 % lr-AdamW/pg2‚8.17730758229248e-05 trainer/global_step‚14499 % _timestamp‚1.7843528081898022e+09ô§Õ*Í·-Ç % lr-AdamW/pg2‚8.17730758229248e-05 trainer/global_step‚14499 % _timestamp‚1.7843528081898022e+09 _runtime‚12466.883388331 _step‚1192 % lr-AdamW/pg1‚8.17730758229248e-05¨ :T¾ç4¸-/ _wandbruntime‚12466  _runtime‚12466Lõ¤\(¹-P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25399101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15340681939968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56571948 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219835834960896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25399098 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15317281952226 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115011836 _step‚1193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115011838 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15341023279104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2204495153020928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249341342 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134329504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2193151901667328 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2193151884627968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115011836 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134319527 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134329504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2204495170060288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2204495170060288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15340681939968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843528081909134e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2208469524169444 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115011838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31172847 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485413794514944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15317622502116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15341023279104 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219835834960896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2204495153020928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15504744927744 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500918558054912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485413813127168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚12466.884649292 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15504654253568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134319527 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31172586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2193151901667328 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2193151884627968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115011836 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134319527 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134329504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2204495170060288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2204495170060288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15340681939968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843528081909134e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2208469524169444 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115011838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31172847 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485413794514944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15317622502116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15341023279104 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219835834960896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2204495153020928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15504744927744 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500918558054912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485413813127168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚12466.884649292 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15504654253568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134319527 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31172586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219835834960896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25399098 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15317281952226 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115011836 _step‚1193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115011838© ÆõQŠ4»-/ _wandbruntime‚12467  _runtime‚12467ÿDû'¼-‰N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15342035913216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚12467.505032543 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2219984447003136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348047 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15505690234368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78435280881125e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2219984447003136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25400843 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2208616688305788 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115019470 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134328579 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1194 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115019470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500953014398464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15318633555950 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2219984088617472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2204642394050560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134338570 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56575504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249358040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15341694566912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56575763 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2193298412339200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15318293005948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134338570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2204642411089920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115019468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2208617045895150 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2193298395299840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15341694566912 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25400846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15342035913216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002u-¾*'½-ŽN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2219984447003136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25400843 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2208616688305788 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115019470 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134328579 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1194 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115019470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500953014398464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15318633555950 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2219984088617472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2204642394050560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134338570 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56575504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249358040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15341694566912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56575763 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2193298412339200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15318293005948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134338570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2204642411089920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115019468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2208617045895150 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2193298395299840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15341694566912 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25400846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15342035913216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15342035913216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚12467.505032543 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2219984447003136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348047 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15505690234368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78435280881125e+09ª ›ÄvÞ4¾-/ _wandbruntime‚12467  _runtime‚12467—yõf¿-Š 0loss/FRt2/assignment_loss‚0.5838904458543529 3loss/SRqqt1/assignment_loss‚0.47133734339819255 3loss/SRqqt2/assignment_loss‚0.44417155708048217 3loss/FBt2/assignment_loss‚0.0017616052689710447 1loss/SRqqt2/detection_loss‚0.6056485697710586 trainer/global_step‚14499 /loss/FRt2/detection_loss‚1.3676548561319748 /loss/FRt1/detection_loss‚0.5371144497864916 0loss/SRqqt1/detection_loss‚0.811949565681414 % _timestamp‚1.7843528088129418e+09 _runtime‚12467.50626495 4loss/FBt1/assignment_loss‚2.5242313755564498e-05 /loss/FBt2/detection_loss‚0.4098075737817207 %loss/total_loss‚7.062209255176359 epoch‚36 _step‚1195 0loss/FBt1/detection_loss‚0.32862890990765986 /loss/FRt1/assignment_loss‚1.500219136200285\-œÀ- /loss/FRt1/assignment_loss‚1.500219136200285 0loss/FRt2/assignment_loss‚0.5838904458543529 3loss/SRqqt1/assignment_loss‚0.47133734339819255 3loss/SRqqt2/assignment_loss‚0.44417155708048217 3loss/FBt2/assignment_loss‚0.0017616052689710447 1loss/SRqqt2/detection_loss‚0.6056485697710586 trainer/global_step‚14499 /loss/FRt2/detection_loss‚1.3676548561319748 /loss/FRt1/detection_loss‚0.5371144497864916 0loss/SRqqt1/detection_loss‚0.811949565681414 % _timestamp‚1.7843528088129418e+09 _runtime‚12467.50626495 4loss/FBt1/assignment_loss‚2.5242313755564498e-05 /loss/FBt2/detection_loss‚0.4098075737817207 %loss/total_loss‚7.062209255176359 epoch‚36 _step‚1195 0loss/FBt1/detection_loss‚0.32862890990765986« e(°äžÁ-‚(:“ µ ìÒ„¨ü“ gpu.0.gpu‚46.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚208.758' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.751599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GR€¢øÂ-‚(:í µ ìÒñ’›– disk.drbd2165.in‚0# disk.dm-0.out‚31096.05419921875( proc.memory.availableMB‚ 400.82421875" disk.nvme0n1.in‚20535.80859375& disk.nvme0n1.out‚31096.05419921875 disk.sda3.out‚ 388.8671875 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0( disk.nvme0c0n1.out‚31096.05419921875 disk.sda.out‚ 388.8671875 cpu‚26.179644828411483 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚17.198467254638672 network.sent‚ 2.3129343e+07) disk./.usagePercent‚39.28266903952199$ disk./.usageGB‚585.3401565551758 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0$ memory_percent‚97.55356311798096 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 network.recv‚ 3.342184e+06 disk.sda1.out‚0! proc.memory.rssMB‚ 2817.796875 disk.rbd1.out‚ 41.2861328125 disk.dm-1.out‚ 388.8671875$ disk.nvme0c0n1.in‚20535.80859375 disk.dm-0.in‚20535.80859375`—>‘Ã-‚(:† Ä ìÒ¸±¹” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s™ìÄ-‚(:á Ä ìÒñߺ• disk.rbd0.in‚0! proc.memory.rssMB‚ 2817.796875 proc.cpu.threads‚17) disk./.usagePercent‚39.28495211471235 disk.dm-1.out‚ 391.21484375 disk.sda.out‚ 391.21484375 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125( proc.memory.availableMB‚ 400.45703125$ disk.nvme0c0n1.in‚20535.80859375 disk.dm-0.in‚20535.80859375! disk.nvme0n1.out‚ 31150.453125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125* proc.memory.percent‚17.198467254638672$ disk./.usageGB‚585.3741760253906 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 31150.453125 disk.rbd0.out‚0 cpu‚26.179872146532276 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.out‚ 31150.453125 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 391.21484375 disk.rbd1.out‚ 41.2861328125$ memory_percent‚97.55580425262451 network.sent‚ 2.3192608e+07 network.recv‚ 3.346741e+06 disk.drbd2165.in‚0" disk.nvme0n1.in‚20535.80859375¶-Ðk4Å-/ _wandbruntime‚12507  _runtime‚12507^ëä‘ÊÆ-Ä & lr-AdamW/pg1‚8.038874110533005e-05 & lr-AdamW/pg2‚8.038874110533005e-05 trainer/global_step‚14549 % _timestamp‚1.7843528486407108e+09 _runtime‚12507.334425688 _step‚1196–zÈÏÇ-É & lr-AdamW/pg1‚8.038874110533005e-05 & lr-AdamW/pg2‚8.038874110533005e-05 trainer/global_step‚14549 % _timestamp‚1.7843528486407108e+09 _runtime‚12507.334425688 _step‚1196¬ 6ŸµØ4È-/ _wandbruntime‚12507  _runtime‚12507íxê¼âÉ-€P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115393438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2227248940322816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56762606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15555846092800 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134783956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134773979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15367898439134 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15391718780928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15391377441792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15391718780928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2200477418221568 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2211857221541888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚1197 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115393436 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2215845674249952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502641097146368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 trainer/global_step‚14549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15555759612928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250167415 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2227248581944320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25486348 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2227248581944320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15391377441792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25486351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502640992054272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250177394 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487085251053568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2227248940322816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2211857221541888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2200477435260928 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56762340 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784352848641919e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115393436 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 J (Ê-…P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250177394 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487085251053568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2227248940322816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2211857221541888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2200477435260928 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56762340 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784352848641919e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115393436 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15555759612928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250167415 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2227248581944320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25486348 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2227248581944320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15391377441792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25486351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502640992054272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184099 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2227397193986560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56766142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134783031 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250194092 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392731415040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56766411 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392731415040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15556779234816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15368909492856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134783031 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134793022 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487118661199872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502675440434688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184099 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2227397193986560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56766142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134783031 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250194092 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392731415040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56766411 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392731415040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15556779234816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15368909492856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134783031 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134793022 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487118661199872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502675440434688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488756688979968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25573601 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135238408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234661687306240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15442072943616 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234661328927744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2219219273023488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843528891977947e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115775036 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2207802951815168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56953282 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488756670367744 trainer/global_step‚14599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15442072943616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15606958794240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003467 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234661328927744 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15442414282752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15606868120064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15418514926042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135228431 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115775038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115775038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2223221466741210 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12547.891515933 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135228431 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504363647774208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2207802968854528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0g£‚ (Ü-†P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504363538487808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15418855475932 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488756688979968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25573601 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135238408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234661687306240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15442072943616 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234661328927744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2219219273023488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843528891977947e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115775036 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2207802951815168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56953282 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488756670367744 trainer/global_step‚14599 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15442072943616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15606958794240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003467 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234661328927744 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15442414282752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15606868120064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15418514926042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135228431 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115775038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115775038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2223221466741210 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12547.891515933 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135228431 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504363647774208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2207802968854528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56953554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1201 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31379953± +k}4Ý-/ _wandbruntime‚12548  _runtime‚12548²Õ_®Þ-ŠN XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2234809940969984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2219366497013760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31381768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504398094550016 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2207949462487040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15443085570560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2234809582584320 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2219366514053120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMon'Fëbitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56957111 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223369346056166 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135247474 trainer/global_step‚14599 $ _timestamp‚1.784352889812678e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15443085570560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115782668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2223368988466804 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56957367 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25575346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚12548.506411087 _step‚1202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25575343 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504397985270784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2234809940969984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31382021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15607976811520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15443426916864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488790117738496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15607886144512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15419866529766 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115782668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2219366514053120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2219366497013760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15443426916864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251020151 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0$N'ß-N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135247474 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115782670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251030144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135237483 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15419525979764 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56957111 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223369346056166 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135247474 trainer/global_step‚14599 $ _timestamp‚1.784352889812678e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15443085570560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115782668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2223368988466804 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56957367 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25575346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚12548.506411087 _step‚1202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25575343 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504397985270784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2234809940969984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31382021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15607976811520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15443426916864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488790117738496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15607886144512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15419866529766 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115782668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2219366514053120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2219366497013760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15443426916864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251020151 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2234809940969984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2219366497013760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31381768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504398094550016 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2207949462487040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15443085570560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2234809582584320 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2219366514053120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1205 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251839519 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2226581307465728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15658017491456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843529295878258e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15492768445440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15657918428672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2242074434289664 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2242074075911168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15493109784576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2242074434289664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57144030 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31483179 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116156636 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116156638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25660848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116156636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2215128485408768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135682883 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2230597974410968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15469471962840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57143769 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2226581324505088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31482921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506086026722816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2242074075911168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135692860 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135682883 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116156638 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2226581324505088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15493109784576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚490428126906368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506086144397824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2230597616821718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2226581307465728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849498„"5ÿî-†P JADeviceStatsM’Â@ªö'onitor.on_train_batch_start/active.small_pool.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15492768445440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25660851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2215128502448128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚14649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1205 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251839519 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2226581307465728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15658017491456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843529295878258e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15492768445440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15657918428672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2242074434289664 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2242074075911168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15493109784576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2242074434289664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57144030 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31483179 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116156636 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116156638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25660848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116156636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2215128485408768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135682883 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2230597974410968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15469471962840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57143769 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2226581324505088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31482921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506086026722816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2242074075911168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135692860 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135682883 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116156638 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2226581324505088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15493109784576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚490428126906368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506086144397824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2230597616821718 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2226581307465728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2230745138547312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866196 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2226728565534720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15494122418688 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15493781072384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506120483580416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15494122418688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135701926 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15659035097600 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1206 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚14649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2226728548495360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2242222329567744 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2226728548495360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866196 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15658946527744 _runtime‚12588.895775551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135701926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25662593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31484894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251856203 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116164268 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856203 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135691935 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31485175 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2226728565534720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2230745496136674 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2215275013120000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116164270 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57147771 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57147487 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490461537052672 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2215274996080640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2242222687953408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¨Â÷C'ñ-N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116164270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135691935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135701926 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15659035097600 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1206 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚14649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2226728548495360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2242222329567744 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2226728548495360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866196 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15658946527744 _runtime‚12588.895775551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135701926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25662593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31484894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251856203 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116164268 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856203 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135691935 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31485175 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2226728565534720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2230745496136674 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2215275013120000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116164270 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57147771 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57147487 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490461537052672 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2215274996080640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2242222687953408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116164268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15493781072384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15470483016674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843529302020142e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2230745138547312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866196 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2226728565534720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15494122418688 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15493781072384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506120483580416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15494122418688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15543805286400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25748101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492099546220544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15520088449748 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2233943375986688 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2233943358947328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492099564832768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2222454019002368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57333788 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116538238 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116538238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252675571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136147312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2249486822894592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1209 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15543463947264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57334049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25748098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2249487181273088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136147312 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2249486822894592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15709064424960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136137335 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136147312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2249486822894592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1209 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15543463947264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57334049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25748098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2249487181273088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136147312 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2249486822894592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15709064424960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136137335 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15543805286400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25748101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492099546220544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15520088449748 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2233943375986688 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2233943358947328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492099564832768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2222454019002368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57333788 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116538238 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169¹ Ð@0¡4ÿ-/ _wandbruntime‚12629  _runtime‚12629÷dÜÜy%€.ŒN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136156378 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2234090599976960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2249635434936832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2234090617016320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492132974979072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136156378 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25749846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252692255 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2234090617016320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136146387 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492132993591296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15709982739456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15544817920512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚14699 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15544817920512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1210 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116545870 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116545868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843529708419476e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15544476574208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2249635076551168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15544476574208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238121646217182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2222600529674240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚12629.535708758 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15520758953580 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116545870 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15521099503582 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2234090599976960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57337530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31587687 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2222600546713600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57337814 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2249635434936832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsò i™Monitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15710085989376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920L^ž`'.‘N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2234090617016320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492132974979072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136156378 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25749846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252692255 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2234090617016320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136146387 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492132993591296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15709982739456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15544817920512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚14699 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15544817920512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1210 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116545870 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116545868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843529708419476e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15544476574208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2249635076551168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15544476574208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238121646217182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2222600529674240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚12629.535708758 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15520758953580 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116545870 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15521099503582 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2234090599976960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57337530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31587687 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2222600546713600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57337814 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2249635434936832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15710085989376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136156378 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2234090599976960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2249635434936832º ãÛ»,4‚./ _wandbruntime‚12629  _runtime‚12629 ïsƒ.Š /loss/FRt1/detection_loss‚0.5246469770105922 1loss/SRqqt1/detection_loss‚0.8115313435362658 epoch‚36 _step‚1211 2loss/SRqqt2/assignment_loss‚0.4404952220246783 3loss/FBt2/assignment_loss‚6.033213609452388e-05 /loss/FBt1/detection_loss‚0.5822008571083036 trainer/global_step‚14699 _runtime‚12629.538735431 0loss/FRt1/assignment_loss‚1.6076087359450917 /loss/FRt2/assignment_loss‚0.672538999426231 /loss/FRt2/detection_loss‚1.1464505012902302 /loss/FBt2/detection_loss‚0.4030135829245954 &loss/total_loss‚7.1918585703626645 % _timestamp‚1.7843529708454573e+09 2loss/SRqqt1/assignment_loss‚0.4745123911978811 4loss/FBt1/assignment_loss‚2.4750709163849174e-05 1loss/SRqqt2/detection_loss‚0.5287748770535372(Ô-V„. _runtime‚12629.538735431 0loss/FRt1/assignment_loss‚1.6076087359450917 /loss/FRt2/assignment_loss‚0.672538999426231 /loss/FRt2/detection_loss‚1.1464505012902302 /loss/FBt2/detection_loss‚0.4030135829245954 &loss/total_loss‚7.1918585703626645 % _timestamp‚1.7843529708454573e+09 2loss/SRqqt1/assignment_loss‚0.4745123911978811 4loss/FBt1/assignment_loss‚2.4750709163849174e-05 1loss/SRqqt2/detection_loss‚0.5287748770535372 /loss/FRt1/detection_loss‚0.5246469770105922 1loss/SRqqt1/detection_loss‚0.8115313435362658 epoch‚36 _step‚1211 2loss/SRqqt2/assignment_loss‚0.4404952220246783 3loss/FBt2/assignment_loss‚6.033213609452388e-05 /loss/FBt1/detection_loss‚0.5822008571083036 trainer/global_step‚14699» éIµ4ž….‚(:“ Ë¡ìÒé结 gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úþ "ì†.‚(:á Ë¡ìÒ„µý•$ disk./.usageGB‚585.3345069885254 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125" disk.nvme0c0n1.in‚ 20778.390625 disk.dm-1.out‚ 401.171875! proc.memory.rssMB‚ 2817.796875 network.recv‚ 3.38092e+06 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.sda2.out‚0& disk.nvme0n1.out‚31222.71044921875 disk.sda3.out‚ 401.171875 disk.rbd0.out‚0 cpu‚26.182098949819864 proc.cpu.threads‚17 network.sent‚ 2.3412171e+07 disk.sda.out‚ 401.171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚31222.71044921875 disk.nvme0n1.in‚ 20778.390625 disk.drbd2165.out‚0$ memory_percent‚97.55773544311523* proc.memory.percent‚17.198467254638672 disk.sda2.in‚0& proc.memory.availableMB‚ 400.140625) disk./.usagePercent‚39.28228989236378 disk.dm-0.in‚ 20778.390625# disk.dm-0.out‚31222.71044921875 disk.rbd1.out‚ 41.6611328125 disk.sda1.out‚0 disk.drbd2165.in‚0z$nN“‡.‚(:ˆ Ú¡ìÒ“×ë” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.129' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T¾mùôˆ.‚(:é Ú¡ìÒõ¨ƒ– network.recv‚ 3.385366e+06( disk.nvme0c0n1.out‚31241.64990234375& disk.nvme0n1.out‚31241.64990234375( proc.memory.availableMB‚ 400.31640625 network.sent‚ 2.3475561e+07 disk.dm-0.in‚ 20778.390625 disk.sda2.out‚0 disk.nvme0n1.in‚ 20778.390625 disk.drbd2165.out‚0! proc.memory.rssMB‚ 2817.796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 403.32421875 disk.rbd1.out‚ 41.6611328125 disk.dm-1.out‚ 403.32421875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.1328125* proc.memory.percent‚17.198467254638672 disk.sda.out‚ 403.32421875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.55666255950928 cpu‚26.18227775022693) disk./.usagePercent‚39.28261783801514$ disk./.usageGB‚585.3393936157227" disk.nvme0c0n1.in‚ 20778.390625# disk.dm-0.out‚31241.64990234375 proc.cpu.threads‚178i ¯“‰.‚(:ˆ é¡ìÒþ³©“ gpu.0.gpu‚46.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B#ÛèŠ.‚(:Ý é¡ìÒÏŽš– network.sent‚ 2.34825e+07& disk.nvme0c0n1.out‚31252.982421875 disk.dm-0.in‚ 20778.390625 disk.sda.out‚ 405.3984375 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125! proc.memory.rssMB‚ 2817.796875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 20778.390625! disk.dm-0.out‚31252.982421875 disk.sda3.out‚ 405.3984375$ memory_percent‚97.55663871765137) disk./.usagePercent‚39.28279397119871 disk.rbd0.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 41.7236328125* proc.memory.percent‚17.198467254638672 disk.sda1.out‚0$ disk.nvme0n1.out‚31252.982421875 disk.drbd2165.in‚0$ disk./.usageGB‚585.3420181274414 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 405.3984375 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.nvme0n1.in‚ 20778.390625' proc.memory.availableMB‚ 400.3203125 cpu‚26.18257379328981 proc.cpu.threads‚17 network.recv‚ 3.389141e+06 disk.dm-1.in‚ 0.1328125©ý 94‹./ _wandbruntime‚12669  _runtime‚12669;YºÊŒ.Ä & lr-AdamW/pg1‚7.494486709152879e-05 & lr-AdamW/pg2‚7.494486709152879e-05 trainer/global_step‚14749 % _timestamp‚1.7843530106517744e+09 _runtime‚12669.345366811 _step‚1212™ëñHÏ.É & lr-AdamW/pg1‚7.494486709152879e-05 & lr-AdamW/pg2‚7.494486709152879e-05 trainer/global_step‚14749 % _timestamp‚1.7843530106517744e+09 _runtime‚12669.345366811 _step‚1212¼ ôÌ-m4Ž./ _wandbruntime‚12669  _runtime‚12669å+®O(.P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31689428 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15760114034176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229779569635328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594500788224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2256899928256512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136591787 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2241305427468288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594500788224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241305427468288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2245350274571984 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493771002759168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2256899569878016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116919836 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241305410428928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594159449088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2256899569878016 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241305410428928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509531209564672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1213 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116919836 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241305410428928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594159449088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2256899569878016 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241305410428928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509531209564672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1213 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31689428 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15760114034176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229779569635328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594500788224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2256899928256512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136591787 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2241305427468288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594500788224 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241305427468288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2245350274571984 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493771002759168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2256899569878016½ %ùwI4‘./ _wandbruntime‚12670  _runtime‚12670Y<æœ'’.ŠN H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2245497438708328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15571715990490 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15595172076032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2257048181920256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31691164 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493804431517696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2241452668497920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241452651458560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843530114636383e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2229926080307200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2245497796297690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116927468 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600839 trainer/global_step‚14749 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57528257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15595172076032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509565664323584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253528307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136610830 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57528533 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚12670.157437324‹s%æ'“.N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15595172076032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509565664323584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253528307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136610830 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57528533 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚12670.157437324 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2245497438708328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15571715990490 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15595172076032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2257048181920256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31691164 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493804431517696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2241452668497920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241452651458560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843530114636383e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2229926080307200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2245497796297690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116927468 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600839 trainer/global_step‚14749 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57528257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888¾ xÈP4”./ _wandbruntime‚12670  _runtime‚12670½Éƶ •.‡ _runtime‚12670.15928393 _step‚1215 /loss/FRt1/assignment_loss‚1.482035607632908 3loss/SRqqt1/assignment_loss‚0.48188722558025626 3loss/FBt2/assignment_loss‚0.0008840009701868797 0loss/FRt2/assignment_loss‚0.7888208060299844 2loss/SRqqt2/assignment_loss‚0.4583245485970958 2loss/FBt1/assignment_loss‚0.003906213149761922 /loss/FRt1/detection_loss‚0.5703682553320397 /loss/FRt2/detection_loss‚1.1311395358364829 %loss/total_loss‚7.402038655530973 1loss/SRqqt2/detection_loss‚0.5803592151171789 /loss/FBt2/detection_loss‚0.3395705825112113 trainer/global_step‚14749 % _timestamp‚1.7843530114659622e+09 epoch‚36 1loss/SRqqt1/detection_loss‚0.9151018001200435 /loss/FBt1/detection_loss‚0.6496408646538233îã}–.Œ 0loss/FRt2/assignment_loss‚0.7888208060299844 2loss/SRqqt2/assignment_loss‚0.4583245485970958 2loss/FBt1/assignment_loss‚0.003906213149761922 /loss/FRt1/detection_loss‚0.5703682553320397 /loss/FRt2/detection_loss‚1.1311395358364829 %loss/total_loss‚7.402038655530973 1loss/SRqqt2/detection_loss‚0.5803592151171789 /loss/FBt2/detection_loss‚0.3395705825112113 trainer/global_step‚14749 % _timestamp‚1.7843530114659622e+09 epoch‚36 1loss/SRqqt1/detection_loss‚0.9151018001200435 /loss/FBt1/detection_loss‚0.6496408646538233 _runtime‚12670.15928393 _step‚1215 /loss/FRt1/assignment_loss‚1.482035607632908 3loss/SRqqt1/assignment_loss‚0.48188722558025626 3loss/FBt2/assignment_loss‚0.0008840009701868797¿ Öøaž—.‚(:“ ø¡ìÒëÞš• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Wéàܘ.‚(:Ñ ø¡ìÒà™Â– disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 31271.796875 disk.sda.in‚ 0.1328125 disk.sda.out‚ 408.90625 disk.sda1.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0$ memory_percent‚97.55711555480957' proc.memory.availableMB‚ 400.2421875* proc.memory.percent‚17.198467254638672$ disk./.usageGB‚585.3377914428711 disk.dm-0.out‚ 31271.796875 disk.sda2.out‚0 disk.dm-1.out‚ 408.90625! proc.memory.rssMB‚ 2817.796875 network.sent‚ 2.3485422e+07 disk.sda2.in‚0 disk.sda3.out‚ 408.90625" disk.nvme0c0n1.in‚ 20778.390625 proc.cpu.threads‚17) disk./.usagePercent‚39.28251031485077 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 41.7236328125 disk.sda1.in‚0 disk.dm-0.in‚ 20778.390625 cpu‚26.182789799665002! disk.nvme0n1.out‚ 31271.796875 disk.drbd2165.out‚0 network.recv‚ 3.392508e+06 disk.nvme0n1.in‚ 20778.390625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.13281250†5“™.‚(:ˆ ‡¢ìÒ³¼ÿ” gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚256.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p(:îš.‚(:ã ‡¢ìÒê‰ó– disk.nvme0n1.in‚ 20860.015625 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.198467254638672 disk.rbd1.out‚ 41.7861328125 disk.rbd0.out‚0 disk.sda1.in‚0( disk./.usagePercent‚39.2828208519898" disk.nvme0c0n1.in‚ 20860.015625( disk.nvme0c0n1.out‚31285.05712890625# disk.dm-0.out‚31285.05712890625 disk.sda.out‚ 410.7109375 disk.sda2.in‚0 disk.sda3.out‚ 410.7109375 network.recv‚ 3.397027e+06 disk.sda3.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125 cpu‚26.18302466669477 network.sent‚ 2.3548589e+07 disk.dm-0.in‚ 20860.015625 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125! proc.memory.rssMB‚ 2817.796875 disk.drbd2165.in‚0 disk.dm-1.out‚ 410.7109375 disk.drbd2165.out‚0$ memory_percent‚97.55582809448242$ disk./.usageGB‚585.3424186706543& disk.nvme0n1.out‚31285.05712890625& proc.memory.availableMB‚ 400.453125 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125¦U 5“›.‚(:ˆ –¢ìÒ‚” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖØ mêœ.‚(:ß –¢ìÒ¡Ö„—! proc.memory.rssMB‚ 2817.796875' disk./.usagePercent‚39.282313957072# disk.dm-0.out‚31299.52001953125 disk.sda.in‚ 0.1328125 disk.sda1.in‚0* proc.memory.percent‚17.198467254638672 cpu‚26.183220786882412 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 413.90625( proc.memory.availableMB‚ 400.45703125 proc.cpu.threads‚17& disk.nvme0n1.out‚31299.52001953125 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0$ memory_percent‚97.55580425262451 network.recv‚ 3.400318e+06" disk.nvme0c0n1.in‚ 20860.015625 disk.sda2.out‚0 disk.nvme0n1.in‚ 20860.015625 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 41.7861328125 network.sent‚ 2.3553451e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 413.90625$ disk./.usageGB‚585.3348655700684( disk.nvme0c0n1.out‚31299.52001953125 disk.dm-0.in‚ 20860.015625 disk.sda3.out‚ 413.90625¢ñC4./ _wandbruntime‚12709  _runtime‚127093»ËÉž.à & lr-AdamW/pg1‚7.360770186734928e-05 & lr-AdamW/pg2‚7.360770186734928e-05 trainer/global_step‚14799 $ _timestamp‚1.784353051254691e+09 _runtime‚12709.948196431 _step‚1216fÇé(Ο.È & lr-AdamW/pg1‚7.360770186734928e-05 & lr-AdamW/pg2‚7.360770186734928e-05 trainer/global_step‚14799 $ _timestamp‚1.784353051254691e+09 _runtime‚12709.948196431 _step‚1216À “ˆÎ4 ./ _wandbruntime‚12709  _runtime‚12709ÙÚŒP?¡.P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚14799 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2264312675239936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843530512558491e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137046239 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XEDeviceStatsMonito‰ÙŠÈ!r.on_train_batch_start/active_bytes.small_pool.freed‚15644854950912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137056216 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2264312316861440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2248667478949888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57715244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357654 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2237105103228928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2264312675239936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15620980873674 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254347675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511253696607744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495442422073344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117301438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2264312316861440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15621321423564 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2237105086189568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2248667461910528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31792646`Èä (¢.†P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117301438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2264312316861440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15621321423564 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2237105086189568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2248667461910528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31792646 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚14799 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1217 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2264312675239936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843530512558491e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137046239 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15644854950912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137056216 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2264312316861440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2248667478949888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57715244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357654 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2237105103228928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2264312675239936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15620980873674 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254347675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511253696607744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495442422073344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Á ªg}†4£./ _wandbruntime‚12710  _runtime‚12710ùpáö'¤.ŠN K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117309068 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137065282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511288254232576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2252873588788836 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117309070 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117309070 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254364359 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2237251596861440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117309068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31794939 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15645867577856 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2248814702940160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137055291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31794675 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15812384788480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2264460570518016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57719285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364359 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495475869444096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2264460928903680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2237251613900800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15622332477398 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57719018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15812292024320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14799 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2264460928903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25924346 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1218 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15645867577856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137065282 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495475850831872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15646208924160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25924343 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15621991927396 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2237251613900800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15622332477398 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57719018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15812292024320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14799 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2264460928903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25924346 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1218 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15645867577856 SEDevice°ÉåPStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137065282 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495475850831872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15646208924160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25924343 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15621991927396 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15645867577856 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2248814702940160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137055291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31794675 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15812384788480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2264460570518016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57719285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364359 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495475869444096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2264460928903680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6856187582015991 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7193581461906433 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5072376132011414 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7636190056800842 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938676953315735 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +jet/accuracy_1_of_1‚0.33076682686805725 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12093064934015274 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6153082251548767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7163420915603638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22549177706241608 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15017859637737274 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25229382514953613 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22419674694538116 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7022514343261719 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20677316188812256 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7571930885314941 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6415034532546997 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8330309987068176 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035060279071331024 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7698644995689392 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047004733234643936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01775858737528324 *jet/accuracy_2_of_4‚0.7645362019538879 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8597316741943359 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2627851068973541 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5803254246711731 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2336428016424179 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7813452482223511 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5659662485122681 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.717553973197937 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_1_of_4‚0.9859188795089722 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8850409984588623 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02786402963101864 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2514795958995819 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696900606155396 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7017512321472168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7328895330429077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6403048634529114 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949389696121216 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954434633255005 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21996216475963593 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6869845986366272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041221365332603455 *jet/accuracy_2_of_2‚0.1259457767009735 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚1220 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5729925036430359 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995517373085022 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652928352356 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028150470927357674 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2702810764312744 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9178666472434998 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382841944694519 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485219597816467 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2077638953924179 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +particle/specificity‚0.8982464075088501 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01775858737528324 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6258137822151184 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.626069962978363 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5088268518447876 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.5934485793113708 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21948003768920898 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2292553335428238 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7977128624916077 ,jet/accuracy_3_of_4‚0.040883906185626984 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884835481643677 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6398743391036987 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25084298849105835 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3696783185005188 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11740589141845703 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035197194665670395 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7103136777877808 )jet/accuracy_1_of_2‚0.770240068435669 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7905378937721252 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9968254566192627 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32060521841049194 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6649633049964905 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8597316741943359 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948610663414001 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972004890441895 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5239769220352173 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24333277344703674 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20695239305496216 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9482590556144714 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 _runtime‚12728.56147827 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13927075266838074 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.626069962978363 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 &particle/f_score‚0.603850245475769 epoch‚36 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9478237628936768 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17593376338481903 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7200365662574768 *jet/accuracy_2_of_3‚0.5033984184265137 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21532469987869263 particle/accuracy_1_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚14799 $ _timestamp‚1.784353069867268e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025040950626134872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7733215093612671 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8857189416885376 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3581686019897461 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7322686314582825 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7430079579353333 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10672948509454727 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3662070631980896 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5050133466720581 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08995687961578369€&½Ø­.‰{ 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8850409984588623 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02786402963101864 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2514795958995819 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696900606155396 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7017512321472168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7328895330429077 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6403048634529114 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949389696121216 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967567920684814 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954434633255005 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976592063903809 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21996216475963593 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6869845986366272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041221365332603455 *jet/accuracy_2_of_2‚0.1259457767009735 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967567920684814 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚1220 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5729925036430359 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995517373085022 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652928352356 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028150470927357674 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2702810764312744 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9178666472434998 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8382841944694519 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485219597816467 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2077638953924179 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +particle/specificity‚0.8982464075088501 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01775858737528324 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6258137822151184 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.626069962978363 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5088268518447876 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.5934485793113708 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21948003768920898 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2292553335428238 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7977128624916077 ,jet/accuracy_3_of_4‚0.040883906185626984 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884835481643677 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6398743391036987 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25084298849105835 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3696783185005188 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11740589141845703 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035197194665670395 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7103136777877808 )jet/accuracy_1_of_2‚0.770240068435669 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7905378937721252 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9968254566192627 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32060521841049194 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6649633049964905 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8597316741943359 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948610663414001 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972004890441895 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5239769220352173 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 particle/accuracy_1_of_4‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24333277344703674 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20695239305496216 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9482590556144714 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 _runtime‚12728.56147827 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13927075266838074 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.626069962978363 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 &particle/f_score‚0.603850245475769 epoch‚36 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9478237628936768 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17593376338481903 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7200365662574768 *jet/accuracy_2_of_3‚0.5033984184265137 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21532469987869263 particle/accuracy_1_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚14799 $ _timestamp‚1.784353069867268e+09 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025040950626134872 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7733215093612671 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8857189416885376 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3581686019897461 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SR¿3k·qqt*FBt/SRqqt_purity‚0.7322686314582825 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7430079579353333 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10672948509454727 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3662070631980896 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5050133466720581 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08995687961578369 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26204031705856323 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6033816933631897 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7817838788032532 +jet/accuracy_3_of_3‚0.04073939844965935 (particle/accuracy‚0.8327908515930176 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6734809279441833 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12479279935359955 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚7.7033787811640275 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6329878568649292 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2952825129032135 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6657065153121948 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024861596524715424 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19010189175605774 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.800278902053833 particle/accuracy_3_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25070470571517944 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19830118119716644 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8857189416885376 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07859616726636887 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6659184694290161 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6267711520195007 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25070470571517944 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7868320941925049 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3020707666873932 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18947483599185944 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0418279655277729 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6856187582015991 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7193581461906433 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5072376132011414 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7636190056800842 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938676953315735 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 +jet/accuracy_1_of_1‚0.33076682686805725 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12093064934015274 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6153082251548767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7163420915603638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22549177706241608 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.15017859637737274 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25229382514953613 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22419674694538116 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7022514343261719 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20677316188812256 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7571930885314941 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6415034532546997 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8330309987068176 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035060279071331024 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7698644995689392 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047004733234643936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01775858737528324 *jet/accuracy_2_of_4‚0.7645362019538879 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8597316741943359 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2627851068973541 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5803254246711731 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2336428016424179 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7813452482223511 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5659662485122681 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.717553973197937 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_1_of_4‚0.9859188795089722Ä Z “®.‚(:ˆ ´¢ìÒè” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õeà¯.‚(:Õ ´¢ìÒÐÝ– disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125$ disk./.usageGB‚585.3352355957031 disk.dm-0.in‚ 20860.015625* proc.memory.percent‚17.214536666870117 network.recv‚ 3.408239e+06 disk.sda3.out‚ 415.1328125 disk.drbd2165.in‚0" disk.nvme0c0n1.out‚ 31308.09375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 415.1328125 network.sent‚ 2.3621687e+07 disk.sda1.in‚0 disk.dm-0.out‚ 31308.09375 disk.sda2.in‚0) disk./.usagePercent‚39.28233878980282 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 20860.015625$ memory_percent‚97.57506847381592( proc.memory.availableMB‚ 397.30078125" proc.memory.rssMB‚ 2820.4296875" disk.nvme0c0n1.in‚ 20860.015625 cpu‚26.18856954249314 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 41.8486328125 disk.dm-1.out‚ 415.1328125 disk.sda2.out‚0 disk.nvme0n1.out‚ 31308.09375§•A‘°.‚(:† âìÒ×Óþ• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0--Á0ð±.‚(:å âìÒÿõ´– network.sent‚ 2.3679302e+07 network.recv‚ 3.412657e+06# disk.dm-0.out‚31312.14404296875& disk.nvme0n1.out‚31312.14404296875 cpu‚26.188838643908042* disk./.usagePercent‚39.282126559556936 disk.sda.out‚ 416.1328125 disk.sda2.in‚0 disk.nvme0n1.in‚ 20954.84375" proc.memory.rssMB‚ 2820.4296875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 20954.84375$ disk./.usageGB‚585.3320732116699 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 416.1328125* proc.memory.percent‚17.214536666870117 disk.rbd0.out‚0( disk.nvme0c0n1.out‚31312.14404296875' proc.memory.availableMB‚ 397.5703125 disk.sda3.out‚ 416.1328125 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚97.57342338562012 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 41.9111328125 disk.sda.in‚ 0.1328125 disk.dm-1.in‚ 0.1328125! disk.nvme0c0n1.in‚ 20954.84375Á?)@².‚(:’ Ò¢ìÒ¾Ÿ• gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ*Ëå³.‚(:Ú Ò¢ìҕ祖 disk.nvme0n1.in‚ 20954.84375 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 20954.84375& proc.memory.availableMB‚ 397.734375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 417.9140625* proc.memory.percent‚17.214536666870117 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 41.9111328125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.57242202758789" disk./.usageGB‚585.33056640625 disk.sda2.in‚0& disk.nvme0c0n1.out‚31323.869140625! disk.dm-0.out‚31323.869140625 cpu‚26.18902808808431 network.recv‚ 3.415454e+06 disk.sda3.in‚ 0.1328125 disk.dm-1.out‚ 417.9140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 417.9140625 disk.rbd0.in‚0 network.sent‚ 2.368335e+07* disk./.usagePercent‚39.282025436580916$ disk.nvme0n1.out‚31323.869140625 disk.dm-0.in‚ 20954.84375" proc.memory.rssMB‚ 2820.4296875¨Ÿ‰4´./ _wandbruntime‚12768  _runtime‚12768ù_[çʵ.Ä % _timestamp‚1.7843531098729026e+09 _runtime‚12768.566491084 _step‚1221 & lr-AdamW/pg1‚7.228023064445508e-05 & lr-AdamW/pg2‚7.228023064445508e-05 trainer/global_step‚14849ûµmÙ϶.É & lr-AdamW/pg1‚7.228023064445508e-05 & lr-AdamW/pg2‚7.228023064445508e-05 trainer/global_step‚14849 % _timestamp‚1.7843531098729026e+09 _runtime‚12768.566491084 _step‚1221Å \! 4·./ _wandbruntime‚12768  _runtime‚127683´Du(¸.€P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2273672363700736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15713283658240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137571013 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31926706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1222 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137580990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255374715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2257959080042496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26012371 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15880728294912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15880639717888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117803702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2246339300036608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2273672722079232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497129790711808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57939346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137580990 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257959097081856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2257959097081856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2273672363700736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31926975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2262028957672309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137580990 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257959097081856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2257959097081856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2273672363700736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31926975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2262028957672309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2273672363700736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15713283658240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137571013 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31926706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1222 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137580990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255374715 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2257959080042496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26012371 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15880728294912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15880639717888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117803702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2246339300036608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2273672722079232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497129790711808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Æ ÔŒ™W4º./ _wandbruntime‚12769  _runtime‚12769ÙA|''».‹N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690328139533 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497163219470336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2273820975742976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391399 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2273820617357312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2246485810708480 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2258106338111488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255391399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31929038 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137590056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2258106321072128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137590056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15714637631488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2258106338111488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2262176121808653 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117811334 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15714637631488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15714296285184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2246485793669120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26014113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57942879 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117811336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2262176479398015 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513044967371776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚14849 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚12769.182825367 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255401392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 _step‚1223HN'¼.N QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26014113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57942879 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117811336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2262176479398015 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513044967371776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚14849 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚12769.182825367 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255401392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 _step‚1223 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690328139533 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497163219470336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2273820975742976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391399 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2273820617357312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2246485810708480 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2258106338111488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255391399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31929038 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137590056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2258106321072128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137590056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15714637631488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2258106338111488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2262176121808653 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117811334 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15714637631488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15714296285184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2246485793669120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15763979160064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514732933080064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2281085469062656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2253664833630208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15739933572719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58129807 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138035442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118185302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26099618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15931723054080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2265321148563456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138025465 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1226 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281085110684160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2253664816590848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281085110684160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2253664816590848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15763979160064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514732933080064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2281085469062656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2253664833630208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15739933572719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58129807 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138035442 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118185302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26099618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15931723054080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2265321148563456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138025465 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1226 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15765333133312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15932832793600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15740944626441 _runtime‚12809.854741808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138034517 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118192936 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118192934 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15764991787008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118192936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15764991787008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 trainer/global_step‚14899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138044508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514767380911104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2269552271889161 % _timestamp‚1.7843531511610172e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2269552629478523 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498834657396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2265468389593088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26101366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32032259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2265468372553728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26101363 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2265468389593088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58133625 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498834638784512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15765333133312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2281233722726400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138044508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15741285176443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58133364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514767490190336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237444 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138034517 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2253811327262720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256227451 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15932742126592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1227 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15765333133312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15932832793600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15740944626441 _runtime‚12809.854741808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138034517 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118192936 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118192934 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15764991787008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118192936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15764991787008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 trainer/global_step‚14899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138044508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514767380911104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2269552271889161 % _timestamp‚1.7843531511610172e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2269552629478523 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498834657396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2265468389593088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26101366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32032259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2265468372553728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26101363 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2265468389593088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58133625 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498834638784512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15765333133312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2281233722726400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138044508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15741285176443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58133364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514767490190336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237444 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138034517 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2253811327262720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256227451 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15932742126592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1227 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2288497857667584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2288498216046080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2272683183005696 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500472666564608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118566904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138489894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1230 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15814674661888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2288498216046080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118566902 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2260990367223808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138489894 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58320168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15790550059627 _runtime‚12849.606632116 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15815016001024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138479917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138479917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516455540985856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186871 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2272683200045056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15982775358464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2260990350184448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2272683200045056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2272683183005696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26186868 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118566904 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15814674661888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257046819 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2288497857667584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2288498216046080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2272683183005696 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500472666564608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118566904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138489894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1230 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15814674661888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2288498216046080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118566902 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2260990367223808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138489894 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58320168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15790550059627 _runtime‚12849.606632116 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15815016001024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138479917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138479917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Î ûIV4Ü./ _wandbruntime‚12850  _runtime‚12850_$ÂdÞÝ.‹N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2261136877895680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118574536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12850.219765449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2272830441074688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500506076710912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2272830441074688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32135331 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15791561113349 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257063503 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58323666 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2288646469709824 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15983802791424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118574534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15816028635136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 %uï^«3 _timestamp‚1.7843531915260255e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257073496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2272830424035328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15816028635136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚14949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32135053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2288646111324160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138488969 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138498960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26188613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15983889264128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15815687288832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15791901663351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118574536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15815687288832 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2276928421969669 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516489984587264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138498960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26188616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58323947 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0òß)â'Þ.N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26188616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58323947 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2261136877895680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118574536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12850.219765449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2272830441074688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500506076710912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2272830441074688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32135331 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15791561113349 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257063503 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58323666 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2288646469709824 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15983802791424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118574534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15816028635136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843531915260255e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257073496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2272830424035328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15816028635136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚14949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32135053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2288646111324160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138488969 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138498960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26188613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15983889264128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15815687288832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15791901663351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118574536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15815687288832 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2276928421969669 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516489984587264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138498960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15865711502848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518177945527808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58510178 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2295910604651008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32236329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16033952420352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2280045251526656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15865711502848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118948504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26274121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2284157407913833 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2284157050324583 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2295910963029504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138944346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2295910604651008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138944346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892850 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚14999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518178056911360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269hö ó0í.†P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15865711502848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518177945527808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58510178 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2295910604651008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32236329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16033952420352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2280045251526656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15865711502848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118948504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26274121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2284157407913833 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2284157050324583 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2295910963029504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138944346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2295910604651008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138944346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892850 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚14999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518178056911360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138934369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15841507096425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15841166546535 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257882871 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118948502 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚502144104491008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚502144085878784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101Ò {ân4î./ _wandbruntime‚12890  _runtime‚12890Z€Æš'ï.ŠN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32238101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909548 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15842177600257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518212501966848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138953412 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15866382790656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2280192475516928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118956134 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138943421 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26275866 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2284304929639539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2280192475516928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502177514637312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2284304572050177 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842518150259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2280192492556288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2280192492556288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502177533249536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15866382790656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257899555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32238369 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16034869661696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2296059216693248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118956134 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16034968717312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118956136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58513964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚518212384299008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2296058858307584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14999 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257899555 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784353232101544e+09 _runtime‚12890.795491368 _step‚1235 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968mÙÅ 'ð.N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16034968717312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118956136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58513964 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚518212384299008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2296058858307584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14999 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257899555 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.784353232101544e+09 _runtime‚12890.795491368 _step‚1235 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32238101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909548 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15842177600257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518212501966848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138953412 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15866382790656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2280192475516928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118956134 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138943421 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26275866 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2284304929639539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2280192475516928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502177514637312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2284304572050177 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842518150259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2280192492556288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2280192492556288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502177533249536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15866382790656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257899555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32238369 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16034869661696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2296059216693248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118956134Ó g°“‡4ñ./ _wandbruntime‚12890  _runtime‚12890̆Nò.ˆ 0loss/FRt1/assignment_loss‚1.6047091846822075 /loss/FRt1/detection_loss‚0.5655878084835813 2loss/SRqqt1/assignment_loss‚0.4608388092307722 2loss/FBt2/assignment_loss‚0.013619005123294687 0loss/FBt1/detection_loss‚0.23523504325978295 %loss/total_loss‚7.232024095255747 epoch‚37 $ _timestamp‚1.784353232105081e+09 0loss/FRt2/assignment_loss‚0.7521045300306058 2loss/FBt1/assignment_loss‚0.004295162975494487 /loss/FRt2/detection_loss‚1.1764106834316443 1loss/SRqqt2/detection_loss‚0.5574064798040749 0loss/FBt2/detection_loss‚0.47887005093400287 _runtime‚12890.798372856 2loss/SRqqt2/assignment_loss‚0.4864274256765021 1loss/SRqqt1/detection_loss‚0.8965199116237843 trainer/global_step‚14999 _step‚1236’köfó. 2loss/SRqqt1/assignment_loss‚0.4608388092307722 2loss/FBt2/assignment_loss‚0.013619005123294687 0loss/FBt1/detection_loss‚0.23523504325978295 %loss/total_loss‚7.232024095255747 epoch‚37 $ _timestamp‚1.784353232105081e+09 0loss/FRt2/assignment_loss‚0.7521045300306058 2loss/FBt1/assignment_loss‚0.004295162975494487 /loss/FRt2/detection_loss‚1.1764106834316443 1loss/SRqqt2/detection_loss‚0.5574064798040749 0loss/FBt2/detection_loss‚0.47887005093400287 _runtime‚12890.798372856 2loss/SRqqt2/assignment_loss‚0.4864274256765021 1loss/SRqqt1/detection_loss‚0.8965199116237843 trainer/global_step‚14999 _step‚1236 0loss/FRt1/assignment_loss‚1.6047091846822075 /loss/FRt1/detection_loss‚0.5655878084835813Ô x1“Ôžô.‚(:“ Ù£ìÒšü®” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_1lîõ.‚(:ã Ù£ìÒ›µñ– disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 420.984375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 420.984375$ disk.nvme0n1.out‚31363.310546875 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚21122.38671875" disk.nvme0n1.in‚21122.38671875' proc.memory.availableMB‚ 398.6640625 proc.cpu.threads‚17& disk.nvme0c0n1.out‚31363.310546875 disk.sda2.out‚0$ disk./.usageGB‚585.3307762145996 disk.dm-0.in‚21122.38671875 disk.sda2.in‚0 network.recv‚ 3.449299e+06) disk./.usagePercent‚39.28203951699529 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 420.984375* proc.memory.percent‚17.214536666870117 network.sent‚ 2.3902639e+07 disk.dm-1.in‚ 0.1328125 disk.rbd1.out‚ 42.2236328125 disk.drbd2165.out‚0$ memory_percent‚97.56674766540527" proc.memory.rssMB‚ 2820.4296875! disk.dm-0.out‚31363.310546875 disk.sda3.in‚ 0.1328125 cpu‚26.19107382203799@ãÂܤö.‚(:ˆ è£ìÒÊ¢• gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0:†Éï.temp‚38.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Xô†ï÷.‚(:ä è£ìÒ©ÙË–% proc.memory.availableMB‚ 398.40625 disk.sda.out‚ 421.28125 disk.sda3.out‚ 421.28125 disk.dm-1.out‚ 421.28125( disk.nvme0c0n1.out‚31367.65185546875( disk./.usagePercent‚39.2820605096131$ disk./.usageGB‚585.3310890197754 disk.dm-0.in‚21122.38671875& disk.nvme0n1.out‚31367.65185546875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 network.recv‚ 3.453659e+06 disk.sda2.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚21122.38671875 disk.sda.in‚ 0.1328125 disk.rbd1.out‚ 42.2861328125 disk.dm-1.in‚ 0.1328125$ memory_percent‚97.56832122802734 network.sent‚ 2.3965748e+07 disk.sda2.in‚0" disk.nvme0n1.in‚21122.38671875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2820.4296875* proc.memory.percent‚17.214536666870117 cpu‚26.191278530826583# disk.dm-0.out‚31367.65185546875 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17?ZN““ø.‚(:ˆ ÷£ìÒâ–ñ“ gpu.0.gpu‚47.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02âgeøù.‚(:í ÷£ìÒÝÒ“–$ disk./.usageGB‚585.3310432434082 cpu‚26.1915605950698 disk.sda.out‚ 421.5234375 disk.sda1.in‚0 disk.drbd2165.in‚0 network.recv‚ 3.457136e+06# disk.dm-0.out‚31371.52001953125 disk.rbd1.out‚ 42.2861328125( proc.memory.availableMB‚ 398.56640625 proc.cpu.threads‚17* disk./.usagePercent‚39.282057437522695 disk.sda.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21122.38671875 disk.sda2.in‚0" disk.nvme0n1.in‚21122.38671875& disk.nvme0n1.out‚31371.52001953125 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0$ memory_percent‚97.56734371185303( disk.nvme0c0n1.out‚31371.52001953125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 421.5234375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2820.4296875* proc.memory.percent‚17.214536666870117 network.sent‚ 2.3970628e+07 disk.dm-0.in‚21122.38671875 disk.dm-1.out‚ 421.5234375§¦®4ú./ _wandbruntime‚12930  _runtime‚129303âgÊÊû.Ä trainer/global_step‚15049 % _timestamp‚1.7843532721367774e+09 _runtime‚12930.830375128 _step‚1237 & lr-AdamW/pg1‚6.706898153152303e-05 & lr-AdamW/pg2‚6.706898153152303e-05¤r!#Ïü.É & lr-AdamW/pg1‚6.706898153152303e-05 & lr-AdamW/pg2‚6.706898153152303e-05 trainer/global_step‚15049 % _timestamp‚1.7843532721367774e+09 _runtime‚12930.830375128 _step‚1237Õ v… 4ý./ _wandbruntime‚12930  _runtime‚12930Y霱(þ.ÿO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32339241 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2287407285968896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15916407004672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15916065665536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚15049 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519900562945024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16084940339200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2303323351634432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15892123583333 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119330104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26361368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139398798 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2291533557994341 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚1238 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58700609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26361371 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519900464144384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119330102 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119330102 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58700609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26361371 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519900464144384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119330102 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119330102 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2303323351634432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15892123583333 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119330104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26361368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139398798 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Ö kÀ¿4€// _wandbruntime‚12931  _runtime‚12931ÁvÒÿ"/‹N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119337734 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745600 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258735607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2303471963676672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1239 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15892794087165 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2287554544037888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139397873 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15917419638784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139397873 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519935019215360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2303471605291008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58704600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2287554544037888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2287554526998528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139407864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32341484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503848971175936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12931.445791264 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58704329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275787945082880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119337736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2303471963676672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275787928043520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26363113 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119337734 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on§í´i_train_batch_end/requested_bytes.all.freed‚2291680722130685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15917078292480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843532727520325e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139407864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2291681079720047 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15917419638784—?åµ'‚/N SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2287554544037888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2287554526998528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139407864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32341484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503848971175936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12931.445791264 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58704329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275787945082880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119337736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2303471963676672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275787928043520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26363113 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119337734 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2291680722130685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15917078292480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843532727520325e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139407864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2291681079720047 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15917419638784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119337734 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745600 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258735607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2303471963676672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1239 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15892794087165 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2287554544037888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139397873 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15917419638784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139397873 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519935019215360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2303471605291008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58704600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0× ñyò:4ƒ// _wandbruntime‚12931  _runtime‚12931ˆdX}„/‰ /loss/FRt1/assignment_loss‚1.432091063169734 0loss/FRt2/assignment_loss‚0.6670638991253717 4loss/FBt1/assignment_loss‚0.00031386209158355426 /loss/FRt1/detection_loss‚0.5608496308308237 1loss/SRqqt2/detection_loss‚0.5537267861144972 _runtime‚12931.448476123 3loss/SRqqt1/assignment_loss‚0.47577299928665573 1loss/SRqqt1/detection_loss‚0.8826390334335785 /loss/FBt1/detection_loss‚0.6798498921637093 %loss/total_loss‚7.261022055062435 % _timestamp‚1.7843532727551901e+09 3loss/FBt2/assignment_loss‚0.0005726094895950664 .loss/FRt2/detection_loss‚1.129213583460244 epoch‚37 trainer/global_step‚15049 _step‚1240 2loss/SRqqt2/assignment_loss‚0.4488300806828431 /loss/FBt2/detection_loss‚0.43009861521379943U¢…/Ž /loss/FRt1/assignment_loss‚1.432091063169734 0loss/FRt2/assignment_loss‚0.6670638991253717 4loss/FBt1/assignment_loss‚0.00031386209158355426 /loss/FRt1/detection_loss‚0.5608496308308237 1loss/SRqqt2/detection_loss‚0.5537267861144972 _runtime‚12931.448476123 3loss/SRqqt1/assignment_loss‚0.47577299928665573 1loss/SRqqt1/detection_loss‚0.8826390334335785 /loss/FBt1/detection_loss‚0.6798498921637093 %loss/total_loss‚7.261022055062435 % _timestamp‚1.7843532727551901e+09 3loss/FBt2/assignment_loss‚0.0005726094895950664 .loss/FRt2/detection_loss‚1.129213583460244 epoch‚37 trainer/global_step‚15049 _step‚1240 2loss/SRqqt2/assignment_loss‚0.4488300806828431 /loss/FBt2/detection_loss‚0.4300986152137994Ø Xø!“†/‚(:ˆ †¤ìÒÓç– gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú¿½/õ‡/‚(:ê †¤ìÒÙ«’—" disk.nvme0n1.in‚21209.12890625 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.56937026977539 cpu‚26.19175887683556 disk.rbd0.out‚0$ disk./.usageGB‚585.3310661315918 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚31377.6826171875 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2820.4296875 network.recv‚ 3.460425e+06 disk.sda1.out‚0 disk.rbd1.out‚ 42.4111328125 disk.dm-1.out‚ 421.77734375& proc.memory.availableMB‚ 398.234375( disk./.usagePercent‚39.2820589735679% disk.nvme0n1.out‚31377.6826171875 disk.sda3.out‚ 421.77734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.rbd0.in‚0* proc.memory.percent‚17.214536666870117 network.sent‚ 2.3975646e+07" disk.dm-0.out‚31377.6826171875 disk.sda.out‚ 421.77734375 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚21209.12890625 disk.dm-0.in‚21209.12890625}Ñଈ/‚(:„ •¤ìÒ•é‘” gpu.0.gpu‚63.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚223.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.66 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ø˜kó‰/‚(:è •¤ìÒ÷ÙÌ–$ disk.nvme0n1.out‚31380.736328125 disk.rbd1.out‚ 42.4111328125 network.sent‚ 2.4038852e+07 network.recv‚ 3.465225e+06 disk.dm-1.in‚ 0.1328125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0$ disk.nvme0c0n1.in‚21209.12890625 disk.sda.out‚ 422.26953125 disk.sda3.in‚ 0.1328125 cpu‚26.192001655466594& disk.nvme0c0n1.out‚31380.736328125 disk.dm-0.in‚21209.12890625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda2.in‚0* disk./.usagePercent‚39.282056669500086 disk.dm-1.out‚ 422.26953125 disk.sda2.out‚0 disk.sda3.out‚ 422.26953125 disk.rbd0.out‚0$ proc.memory.availableMB‚397.6875" proc.memory.rssMB‚ 2820.4296875* proc.memory.percent‚17.214536666870117$ memory_percent‚97.57270812988281! disk.dm-0.out‚31380.736328125$ disk./.usageGB‚585.3310317993164 disk.rbd1.in‚ 2.01953125" disk.nvme0n1.in‚21209.12890625»_C/4Š// _wandbruntime‚12971  _runtime‚12971JHҪɋ/à trainer/global_step‚15099 $ _timestamp‚1.784353312488795e+09 _runtime‚12971.182368477 _step‚1241 & lr-AdamW/pg1‚6.579124658662917e-05 & lr-AdamW/pg2‚6.579124658662917e-05t'B|ÎŒ/È _runtime‚12971.182368477 _step‚1241 & lr-AdamW/pg1‚6.579124658662917e-05 & lr-AdamW/pg2‚6.579124658662917e-05 trainer/global_step‚15099 $ _timestamp‚1.784353312488795e+09Ù h44// _wandbruntime‚12971  _runtime‚12971YÃvä(Ž/P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2294769337450496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15967102506496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26448618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521623087245312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 trainer/global_step‚15099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139853250 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139843273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843533124898324e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119711702 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15967102506496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2298909708074849 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2310736456996352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2310736098617856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2298909350485599 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1242 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16136016227328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119711702 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58891640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15942399520351 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2310736098617856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58891364 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139853250 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚12971.183552933 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119711704 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2294769354489856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119711704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505486961731584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16136106901504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2282966950965248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2294769354489856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2310736098617856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2298909350485599 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1242 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16136016227328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119711702 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58891640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15942399520351 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2310736098617856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58891364 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139853250 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚12971.183552933 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119711704 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2294769354489856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119711704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505486961731584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16136106901504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2282966950965248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2294769354489856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 trainer/global_step‚15099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139853250 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139843273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843533124898324e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119711702 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376Ú ú<ù’4// _wandbruntime‚12971  _runtime‚12971”=6S'‘/‹N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119719336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2294916578480128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16137124804096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2294916595519488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521657533906432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15943410574073 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505520409102336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139852325 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2294916595519488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26450363 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15968115140608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139862316 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58895142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26450366 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843533131031713e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16137034137088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2299057229800555 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571659 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119719336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32444779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2310884352274432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139862316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2283113461637120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259581652 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32445049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15099 _runtime‚12971.796890485 _step‚1243 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15099 _runtime‚12971.796890485 _step‚1243 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15943410574073 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505520409102336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139852325 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2294916595519488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26450363 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15968115140608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139862316 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58895142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26450366 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843533131031713e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16137034137088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2299057229800555 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571659 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119719336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32444779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2310884352274432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139862316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2283113461637120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438Û .Ë4“// _wandbruntime‚12971  _runtime‚12971º6Èâ ”/‡ /loss/FRt2/detection_loss‚1.1845212277660035 1loss/SRqqt2/detection_loss‚0.5942205311370055 %loss/total_loss‚7.462779363115358 trainer/global_step‚15099 1loss/SRqqt1/assignment_loss‚0.477731992888119 4loss/FBt1/assignment_loss‚1.8973482943757907e-05 /loss/FRt1/detection_loss‚0.5316695750695355 1loss/SRqqt1/detection_loss‚0.8882782559134814 /loss/FBt2/detection_loss‚0.4760083288271854 % _timestamp‚1.7843533131070166e+09 _runtime‚12971.80032547 0loss/FRt1/assignment_loss‚1.4726419475073467 0loss/FRt2/assignment_loss‚0.7057814948567737 2loss/SRqqt2/assignment_loss‚0.4994592615663783 /loss/FBt1/detection_loss‚0.6015604973986878 2loss/FBt2/assignment_loss‚0.030887276701897753 epoch‚37 _step‚1244Qܸ_•/Œ /loss/FRt2/detection_loss‚1.1845212277660035 1loss/SRqqt2/detection_loss‚0.5942205311370055 %loss/total_loss‚7.462779363115358 trainer/global_step‚15099 1loss/SRqqt1/assignment_loss‚0.477731992888119 4loss/FBt1/assignment_loss‚1.8973482943757907e-05 /loss/FRt1/detection_loss‚0.5316695750695355 1loss/SRqqt1/detection_loss‚0.8882782559134814 /loss/FBt2/detection_loss‚0.4760083288271854 % _timestamp‚1.7843533131070166e+09 _runtime‚12971.80032547 0loss/FRt1/assignment_loss‚1.4726419475073467 0loss/FRt2/assignment_loss‚0.7057814948567737 2loss/SRqqt2/assignment_loss‚0.4994592615663783 /loss/FBt1/detection_loss‚0.6015604973986878 2loss/FBt2/assignment_loss‚0.030887276701897753 epoch‚37 _step‚1244Ü ÛŽ–/‚(:’ ¤¤ìÒ‘¸«“ gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ß 1ð—/‚(:å ¤¤ìÒÂÇÈ•( disk./.usagePercent‚39.2820507813268& disk.nvme0n1.out‚31385.26025390625" proc.memory.rssMB‚ 2820.4296875 disk.dm-1.in‚ 0.1328125 disk.rbd1.out‚ 42.4736328125 disk.rbd0.in‚0 network.recv‚ 3.468478e+06( disk.nvme0c0n1.out‚31385.26025390625 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17 network.sent‚ 2.4043835e+07# disk.dm-0.out‚31385.26025390625 disk.dm-1.out‚ 422.84375 disk.drbd2165.out‚0 disk.sda.out‚ 422.84375 disk.sda1.out‚0 disk.dm-0.in‚21209.12890625 disk.sda2.out‚0 disk.sda3.out‚ 422.84375 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚26.192188231799875$ disk./.usageGB‚585.3309440612793 disk.sda3.in‚ 0.1328125$ memory_percent‚97.57413864135742& proc.memory.availableMB‚ 397.453125* proc.memory.percent‚17.214536666870117 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚21209.12890625 disk.sda.in‚ 0.1328125 disk.sda2.in‚0" disk.nvme0n1.in‚21209.12890625fጓ˜/‚(:ˆ ³¤ìÒ€ãÑ” gpu.0.gpu‚61.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚245.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä\å™/‚(:Ú ³¤ìÒÖÞ— network.recv‚ 3.473062e+06 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 397.3046875 network.sent‚ 2.4107008e+07 disk.nvme0n1.out‚ 31388.5625 disk.dm-1.out‚ 423.38671875$ disk.nvme0c0n1.in‚21209.12890625 disk.sda3.out‚ 423.38671875 disk.rbd1.out‚ 42.4736328125 disk.sda.in‚ 0.1328125 cpu‚26.19241861312772 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 423.38671875* proc.memory.percent‚17.214536666870117) disk./.usagePercent‚39.28203260479187 disk.sda1.in‚0! disk.nvme0c0n1.out‚ 31388.5625$ memory_percent‚97.57504463195801 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.out‚ 31388.5625 proc.cpu.threads‚17$ disk./.usageGB‚585.3306732177734" disk.nvme0n1.in‚21209.12890625 disk.sda1.out‚0 disk.dm-0.in‚21209.12890625 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2820.4296875—ðT“š/‚(:ˆ ¤ìÒ¼Š–• gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.833' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡|8ù›/‚(:î ¤ìÒ€¿Ì– disk.sda2.out‚0& disk.nvme0n1.out‚31393.38037109375* proc.memory.percent‚17.214536666870117 disk.dm-0.in‚21281.54296875 proc.cpu.threads‚17( proc.memory.availableMB‚ 397.54296875) disk./.usagePercent‚39.28203516486721 disk.rbd1.out‚ 42.5361328125 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0# disk.dm-0.out‚31393.38037109375" disk.nvme0n1.in‚21281.54296875 cpu‚26.192623869206386 network.sent‚ 2.4112105e+07 disk.sda2.in‚0 disk.sda3.out‚ 423.5859375 disk.sda3.in‚ 0.1328125$ memory_percent‚97.57359027862549 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21281.54296875( disk.nvme0c0n1.out‚31393.38037109375" proc.memory.rssMB‚ 2820.4296875 network.recv‚ 3.476164e+06$ disk./.usageGB‚585.3307113647461 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 423.5859375 disk.drbd2165.in‚0 disk.sda.out‚ 423.5859375 disk.rbd1.in‚ 2.01953125Ћb4œ// _wandbruntime‚13011  _runtime‚13011ïuˆšÊ/Ä _step‚1245 & lr-AdamW/pg1‚6.452370767933704e-05 & lr-AdamW/pg2‚6.452370767933704e-05 trainer/global_step‚15149 % _timestamp‚1.7843533530607574e+09 _runtime‚13011.754327411]4I½Ïž/É & lr-AdamW/pg2‚6.452370767933704e-05 trainer/global_step‚15149 % _timestamp‚1.7843533530607574e+09 _runtime‚13011.754327411 _step‚1245 & lr-AdamW/pg1‚6.452370767933704e-05Ý $¹ç4Ÿ// _wandbruntime‚13011  _runtime‚13011¯ÞDh>  /€P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16017798008320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523345560040960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2290292501598208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15993356557149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2302131405971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59081921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16187053193728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_bäŽDÈatch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120093302 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523345452851712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16187141770752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2302131388932096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140297725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507158399657984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16017456669184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2302131388932096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16017798008320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26535868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140307702 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2318148845601280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2302131405971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140307702 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59082190 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16017456669184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120093304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2306285500566107 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977%Ýë— (¡/…P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16017798008320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523345560040960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2290292501598208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15993356557149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2302131405971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59081921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16187053193728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120093302 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523345452851712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16187141770752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2302131388932096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140297725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507158399657984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16017456669184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2302131388932096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16017798008320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26535868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140307702 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2318148845601280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2302131405971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140307702 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59082190 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16017456669184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120093304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2306285500566107 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360Þ ða׈4¢// _wandbruntime‚13012  _runtime‚13012š·A'£/ŠN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140306777 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16018469296128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32548119 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2318297457643520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59085987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32548371 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2306433379881063 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2306433022291701 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15994027060981 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140316768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120100934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120100936 _step‚1247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140306777 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16018469296128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260407711 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140316768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507191847028736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2302278647001088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59085732 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16188161123328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120100934 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2302278629961728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26537616 _runtime‚13012.369561973 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32548119 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2318297457643520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59085987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32548371 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2306433379881063 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2306433022291701 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15994027060981 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocat¨+GWOed_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140316768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120100934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120100936 _step‚1247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140306777 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16018469296128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260407711 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140316768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507191847028736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2302278647001088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59085732 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16188161123328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120100934 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2302278629961728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26537616 _runtime‚13012.369561973 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16068493510144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508829856196608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140752177 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525068084009984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59272560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1250 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325561592584704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16068152171008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2309493457453056 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2297618018152448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261237058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26623121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26623118 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32649442 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2309493440413696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16068493510144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120474904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32649720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140762154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚508829837584384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525067972626432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16238227813376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120474902 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 $ _timestamp‚1.784353393394554e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59272560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1250 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325561592584704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16068152171008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2309493457453056 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2297618018152448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261237058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26623121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26623118 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32649442 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2309493440413696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16068493510144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120474904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32649720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140762154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚508829837584384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525067972626432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16238227813376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120474902 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 $ _timestamp‚1.784353393394554e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16068493510144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508829856196608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1â ¦ût4´// _wandbruntime‚13052  _runtime‚13052£‡§÷µ/‹N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140761229 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2297764‹*^&528824320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16069506144256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508863266342912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16239246633472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843533940092897e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120482534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26624866 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2325709846241280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140771220 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525102531588608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚13052.703015767 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16069164797952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261243763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16044643547889 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525102426503680 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325710204626944 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚15199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2313809529961571 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261243763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120482536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261253756 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32651730 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16069164797952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2309640698482688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2309640681443328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120482534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2297764545863680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120482536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325710204626944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59276312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32651449 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚15199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2313809529961571 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261243763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120482536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261253756 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32651730 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16069164797952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2309640698482688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2309640681443328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120482534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2297764545863680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120482536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325710204626944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59276312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32651449 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508863266342912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16239246633472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843533940092897e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120482534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26624866 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2325709846241280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140771220 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525102531588608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚13052.703015767 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16069164797952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261243763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16044643547889 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525102426503680 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325710204626944 Q&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14075641334056854 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7900804877281189 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.707664966583252 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29957833886146545 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4995463192462921 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17801643908023834 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9219312071800232 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03184787556529045 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7485800981521606 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07621298730373383 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.7590065002441406 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2764899730682373 *jet/accuracy_2_of_2‚0.1252441555261612 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2336205691099167 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982857704162598 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965319633483887 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20902325212955475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6413483619689941 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7869330644607544 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23805010318756104 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7195166349411011 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19596776366233826 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.786929726600647 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980900883674622 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7049475908279419 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12763868272304535 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975374937057495 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6260936260223389 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6255067586898804 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944190979003906 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2603115439414978 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.711567223072052 +jet/accuracy_4_of_4‚0.01670061983168125 6validation_average_jet_accuracy‚0.3688381612300873 trainer/global_step‚15199 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609598875045776 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14537136256694794 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11960207670927048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_3_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6830801963806152 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6602821350097656 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8342215418815613 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7845918536186218 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6643229722976685 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5186610817909241 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882186651229858 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5786679983139038 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5722420811653137 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6376303434371948 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_4_of_4‚0.9447077512741089 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6388692259788513 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.711280882358551 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49484261870384216 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.040446214377880096 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3243560194969177 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2204018086194992 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23058757185935974 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9802970886230469 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6291895508766174 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7770689725875854 (particle/accuracy‚0.8311516046524048 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2272886335849762 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971826672554016 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933008551597595 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8738692402839661 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6842204332351685 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8760325312614441 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21491873264312744 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.602!×ñ¯ 0647883415222 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843534121753545e+09 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7710695862770081 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950392842292786 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6427640318870544 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.706173300743103 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20351211726665497 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21219968795776367 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7257348299026489 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7258397340774536 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045480452477931976 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6483001708984375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11354948580265045 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951696991920471 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21630066633224487 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2603115439414978 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/specificity‚0.8967072367668152 'particle/f_score‚0.6007100939750671 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.760762095451355 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7700896263122559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763542890548706 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.310104638338089 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3679191768169403 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01775858737528324 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26014578342437744 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975374937057495 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0245718564838171 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869582653045654 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7081533074378967 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.043876513838768005 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.04295429587364197 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023151718080043793 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5648801922798157 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9381422400474548 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022988909855484962 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚13070.869507479 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6255067586898804 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26157739758491516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024803372099995613 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2718107998371124 )loss/val_total_loss‚7.713679145672457 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01775858737528324 )jet/accuracy_2_of_3‚0.502988874912262 +particle/sensitivity‚0.5914490222930908 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25238746404647827 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039571236819028854 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7672206163406372 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5340102314949036 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22815561294555664 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27137526869773865 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35607361793518066 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.130973339080810554~½×Ž=¾/ˆ{ 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7869330644607544 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23805010318756104 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7195166349411011 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19596776366233826 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.786929726600647 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980900883674622 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7049475908279419 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12763868272304535 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975374937057495 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6260936260223389 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6255067586898804 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944190979003906 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2603115439414978 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.711567223072052 +jet/accuracy_4_of_4‚0.01670061983168125 6validation_average_jet_accuracy‚0.3688381612300873 trainer/global_step‚15199 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609598875045776 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14537136256694794 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11960207670927048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_3_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6830801963806152 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6602821350097656 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8342215418815613 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7845918536186218 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6643229722976685 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5186610817909241 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882186651229858 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5786679983139038 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5722420811653137 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6376303434371948 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_4_of_4‚0.9447077512741089 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6388692259788513 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.711280882358551 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49484261870384216 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.040446214377880096 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3243560194969177 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2204018086194992 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23058757185935974 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9802970886230469 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6291895508766174 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7770689725875854 (particle/accuracy‚0.8311516046524048 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2272886335849762 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971826672554016 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933008551597595 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8738692402839661 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6842204332351685 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8760325312614441 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21491873264312744 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6020647883415222 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843534121753545e+09 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7710695862770081 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950392842292786 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6427640318870544 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.706173300743103 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20351211726665497 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21219968795776367 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7257348299026489 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7258397340774536 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045480452477931976 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6483001708984375 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11354948580265045 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951696991920471 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21630066633224487 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2603115439414978 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/specificity‚0.8967072367668152 'particle/f_score‚0.6007100939750671 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.760762095451355 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7700896263122559 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763542890548706 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.310104638338089 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3679191768169403 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01775858737528324 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26014578342437744 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975374937057495 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0245718564838171 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869582653045654 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7081533074378967 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.043876513838768005 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.04295429587364197 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023151718080043793 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5648801922798157 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9381422400474548 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022988909855484962 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚13070.869507479 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6255067586898804 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26157739758491516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024803372099995613 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2718107998371124 )loss/val_total_loss‚7.713679145672457 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01775858737528324 )jet/accuracy_2_of_3‚0.502988874912262 +particle/sensitivity‚0.5914490222930908 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25238746404647827 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039571236819028854 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7672206163406372 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5340102314949036 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22815561294555664 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27137526869773865 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35607361793518066 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13097333908081055 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8738692402839661 _step‚1253 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7916309833526611 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6614081263542175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19553668797016144 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.771759033203125 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866988062858582 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 particle/accuracy_3_of_4‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6158768534660339 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.031402599066495895 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2575805187225342 jet/accuracy_1_of_5‚1 epoch‚37 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8420257568359375 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943126440048218 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08786053955554962 *jet/accuracy_1_of_1‚0.3342549800872803 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14075641334056854 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7900804877281189 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.707664966583252 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29957833886146545 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4995463192462921 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17801643908023834 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9219312071800232 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03184787556529045 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7485800981521606 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07621298730373383 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.7590065002441406 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2764899730682373 *jet/accuracy_2_of_2‚0.1252441555261612 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2336205691099167 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982857704162598 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9965319633483887 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20902325212955475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6413483619689941å '+4п/‚(:’ ¥ìÒ‘¾ž” gpu.0.gpu‚22.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.105' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.621000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÞÝÜÀ/‚(:Ñ ¥ìÒ¾£þ• network.sent‚ 2.4253105e+07" disk.nvme0n1.out‚ 31412.0078125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 425.34375 proc.memory.rssMB‚ 2823.0625 disk.sda2.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 21359.765625* proc.memory.percent‚17.230606079101562 cpu‚26.198447199876643 disk.sda.in‚ 0.1328125' proc.memory.availableMB‚ 394.5390625$ disk./.usageGB‚585.3308486938477 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda3.out‚ 425.34375 network.recv‚ 3.494742e+06 disk.nvme0n1.in‚ 21359.765625 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.dm-1.out‚ 425.34375 disk.drbd2165.in‚0# memory_percent‚97.5919246673584) disk./.usagePercent‚39.28204438113845$ disk.nvme0c0n1.out‚ 31412.0078125 disk.dm-0.in‚ 21359.765625 disk.dm-0.out‚ 31412.0078125 disk.rbd1.out‚ 42.6611328125«?5;žÁ/‚(:“ œ¥ìÒ̆æ” gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚253.165' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.632999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïA†ŽïÂ/‚(:ä œ¥ìÒÜЖ– disk.sda.out‚ 425.55859375$ disk./.usageGB‚585.3271064758301 disk.sda.in‚ 0.1328125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 21359.765625 disk.dm-0.in‚ 21359.765625* proc.memory.percent‚17.230606079101562 network.recv‚ 3.498975e+06% disk.nvme0n1.out‚31418.2763671875 disk.rbd1.in‚ 2.01953125' disk.nvme0c0n1.out‚31418.2763671875' proc.memory.availableMB‚ 394.1484375 cpu‚26.198619629862026 disk.sda3.out‚ 425.55859375 disk.rbd1.out‚ 42.7236328125 disk.rbd0.in‚0 network.sent‚ 2.4310703e+07 disk.nvme0n1.in‚ 21359.765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 425.55859375) disk./.usagePercent‚39.28179323774736 disk.rbd0.out‚0" disk.dm-0.out‚31418.2763671875 disk.sda2.in‚0 disk.drbd2165.in‚0$ memory_percent‚97.59430885314941 proc.memory.rssMB‚ 2823.0625 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.sda2.out‚0ãg&€“Ã/‚(:ˆ «¥ìÒ°Çà” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚196.844' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.3688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ü,5ÞÄ/‚(:Ó «¥ìÒ¯Ù• disk.sda3.out‚425.9375 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 proc.memory.rssMB‚ 2823.0625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 21359.765625 disk.rbd0.out‚0 network.sent‚ 2.4315674e+07 disk.sda1.in‚0$ memory_percent‚97.59187698364258* proc.memory.percent‚17.230606079101562" disk.nvme0c0n1.in‚ 21359.765625 disk.rbd1.out‚ 42.7236328125 network.recv‚ 3.502536e+06) disk./.usagePercent‚39.28179349375489$ disk./.usageGB‚585.3271102905273 disk.dm-0.in‚ 21359.765625 disk.dm-1.out‚425.9375! disk.dm-0.out‚31422.634765625 disk.sda1.out‚0& proc.memory.availableMB‚ 394.546875 cpu‚26.19883251353929 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125& disk.nvme0c0n1.out‚31422.634765625 disk.sda.out‚425.9375$ disk.nvme0n1.out‚31422.634765625bU¿4Å// _wandbruntime‚13110  _runtime‚13110ƒ*óÊÆ/Ä & lr-AdamW/pg1‚6.201954296407865e-05 & lr-AdamW/pg2‚6.201954296407865e-05 trainer/global_step‚15249 % _timestamp‚1.7843534522079444e+09 _runtime‚13110.901570842 _step‚1254µ6ÔÏÇ/É & lr-AdamW/pg1‚6.201954296407865e-05 & lr-AdamW/pg2‚6.201954296407865e-05 trainer/global_step‚15249 % _timestamp‚1.7843534522079444e+09 _runtime‚13110.901570842 _step‚1254æ âçÚÌ4È// _wandbruntime‚13110  _runtime‚13110$T³&“É/€P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262254119 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141286928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32783841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510517206222848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120977170 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2334921997802496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136580878336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510517187610624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141286928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16136922217472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚15249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26712888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1255 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120977170 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2318785075585024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120977168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16111968706304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262264098 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322964183666432 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2306852231999488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141276951 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32783588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13110.902882759 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.ìh!˜ssmall_pool.freed‚16307503407616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2322964541255682 % _timestamp‚1.7843534522090657e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120977168 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2334921639424000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59496476 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2334921997802496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16112309256194 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2318785058545664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141276951 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16136922217472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526824781430272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526824691018240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2334921639424000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59496732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2306852214960128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141286928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16136922217472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚15249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26712888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1255 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120977170 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2318785075585024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120977168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16111968706304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262264098 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322964183666432 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2306852231999488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141276951 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32783588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13110.902882759 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16307503407616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2322964541255682 % _timestamp‚1.7843534522090657e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120977168 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2334921639424000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59496476 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2334921997802496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16112309256194 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2318785058545664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141276951 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16136922217472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526824781430272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526824691018240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2334921639424000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59496732 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2306852214960128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262254119 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141286928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32783841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510517206222848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120977170 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2334921997802496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136580878336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510517187610624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16308607376384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141295994 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2306998742671360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262280796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26714633 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2335070251466240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510550634981376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2323111705392026 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32785971 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2318932316614656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26714636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2335069893080576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16112979760026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318932316614656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318932299575296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59500339 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2335070251466240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚13111.518538153 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120984802 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120984800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526859143564288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141295994 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16137593505280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16137593505280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32785706 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2318932299575296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438d3º!Í/N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2318932316614656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26714636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2335069893080576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16112979760026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318932316614656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318932299575296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59500339 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2335070251466240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚13111.518538153 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120984802 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120984800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526859143564288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141295994 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16137593505280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16137593505280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32785706 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2318932299575296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16137934851584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526859242357760 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120984800 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2323112062981388 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16137934851584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141286003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510550616369152 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120984802 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16308607376384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_U‰˜úend/inactive_split.all.peak‚544 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141295994 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2306998742671360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262280796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26714633 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2335070251466240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510550634981376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2323111705392026 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32785971 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088è Cwõ4Î// _wandbruntime‚13111  _runtime‚13111/–Ä Ï/… /loss/FBt1/detection_loss‚0.3156573324860703 /loss/FBt2/detection_loss‚0.4215645571636622 trainer/global_step‚15249 _runtime‚13111.52032668 2loss/SRqqt2/assignment_loss‚0.4394475802982684 2loss/FBt1/assignment_loss‚7.29391401888291e-06 3loss/FBt2/assignment_loss‚0.0005347547128970785 1loss/SRqqt2/detection_loss‚0.6126155579065637 epoch‚38 /loss/FRt1/detection_loss‚0.4912331861637443 %loss/total_loss‚6.963528372759703 $ _timestamp‚1.784353452827011e+09 0loss/FRt2/assignment_loss‚0.7420443733220953 .loss/FRt2/detection_loss‚1.225359691643844 0loss/SRqqt1/detection_loss‚0.863544101554169 _step‚1257 0loss/FRt1/assignment_loss‚1.4492391084322838 3loss/SRqqt1/assignment_loss‚0.40228083516208624¯î–ìÐ/Š 0loss/FRt2/assignment_loss‚0.7420443733220953 .loss/FRt2/detection_loss‚1.225359691643844 0loss/SRqqt1/detection_loss‚0.863544101554169 _step‚1257 0loss/FRt1/assignment_loss‚1.4492391084322838 3loss/SRqqt1/assignment_loss‚0.40228083516208624 /loss/FBt1/detection_loss‚0.3156573324860703 /loss/FBt2/detection_loss‚0.4215645571636622 trainer/global_step‚15249 _runtime‚13111.52032668 2loss/SRqqt2/assignment_loss‚0.4394475802982684 2loss/FBt1/assignment_loss‚7.29391401888291e-06 3loss/FBt2/assignment_loss‚0.0005347547128970785 1loss/SRqqt2/detection_loss‚0.6126155579065637 epoch‚38 /loss/FRt1/detection_loss‚0.4912331861637443 %loss/total_loss‚6.963528372759703 $ _timestamp‚1.784353452827011e+09é rÖˆæÑ/‚(:’ º¥ìÒáëš” gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´à~%ïÒ/‚(:ä º¥ìÒ‹–˜ disk.dm-1.in‚ 0.1328125 cpu‚26.19899464942368 disk.dm-1.out‚ 426.26953125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 42.7861328125 disk.rbd0.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 21440.34375 network.recv‚ 3.505763e+06 disk.dm-0.in‚ 21440.34375# disk.dm-0.out‚31429.23876953125 disk.sda3.out‚ 426.26953125$ memory_percent‚97.59325981140137 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚31429.23876953125 disk.sda1.in‚0' proc.memory.availableMB‚ 394.3203125* proc.memory.percent‚17.230606079101562 network.sent‚ 2.4333464e+07* disk./.usagePercent‚39.282325733418574 disk.nvme0n1.in‚ 21440.34375( disk.nvme0c0n1.out‚31429.23876953125 proc.cpu.threads‚17 disk.sda2.in‚0 proc.memory.rssMB‚ 2823.0625$ disk./.usageGB‚585.3350410461426 disk.sda.out‚ 426.26953125 disk.sda2.out‚0ëÀ:“Ó/‚(:ˆ É¥ìÒݧ¤• gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.847' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.WBêÔ/‚(:ß É¥ìÒ¡¼‘— disk.rbd0.in‚0 disk.dm-0.in‚ 21440.34375( proc.memory.availableMB‚ 394.46484375 proc.memory.rssMB‚ 2823.0625 network.recv‚ 3.51024e+06 disk.sda3.out‚ 427.9921875 disk.sda.out‚ 427.9921875 disk.nvme0n1.in‚ 21440.34375 network.sent‚ 2.4385463e+07 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 21440.34375 disk.sda2.in‚0$ disk./.usageGB‚585.4616050720215 disk.dm-1.out‚ 427.9921875# disk.dm-0.out‚31462.46630859375 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 42.7861328125& disk.nvme0n1.out‚31462.46630859375$ memory_percent‚97.59237766265869 cpu‚26.19922788444812 proc.cpu.threads‚17( disk./.usagePercent‚39.2908195513896 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚31462.46630859375 disk.sda.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.sda1.out‚0* proc.memory.percent‚17.230606079101562•±[º4Õ// _wandbruntime‚13151  _runtime‚13151C^:¥ËÖ/Å ' lr-AdamW/pg2‚6.0783077993932565e-05 trainer/global_step‚15299 $ _timestamp‚1.784353492654734e+09 _runtime‚13151.348391124 _step‚1258 ' lr-AdamW/pg1‚6.0783077993932565e-05Œ² Ð×/Ê ' lr-AdamW/pg1‚6.0783077993932565e-05 ' lr-AdamW/pg2‚6.0783077993932565e-05 trainer/global_step‚15299 $ _timestamp‚1.784353492654734e+09 _runtime‚13151.348391124 _step‚1258ê > 4Ø// _wandbruntime‚13151  _runtime‚13151HÚŸ(Ù/P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32887683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512188644149248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090171 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2326147110027264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263100150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26800138 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26800141 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528547772042240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528547671144448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2326147127066624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59687556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32887418 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16359045607424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚15299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187276380160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141741380 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2342334386407424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187276380160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16162925743102 % _timestamp‚1.7843534926558955e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2314177748553728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚13151.349625745 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2342334744785920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2326147127066624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979—·ÿ (Ú/†P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2314177748553728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚13151.349625745 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2342334744785920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2326147127066624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32887683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512188644149248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090171 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2326147110027264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263100150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26800138 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26800141 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528547772042240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528547671144448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2326147127066624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59687556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32887418 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16359045607424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚15299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187276380160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141741380 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2342334386407424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187276380160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16162925743102 % _timestamp‚1.7843534926558955e+09ë á Œ4Û// _wandbruntime‚13151  _runtime‚131513B8ôÜ/ŠN QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2342482998449664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784353493272062e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16188289007104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2326294351056896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2314324259225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121366400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121366402 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141740455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16163936796936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59691588 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141750446 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2326294368096256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2330488213061896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2330487855472534 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26801886 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚512222054295552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59691311 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚13151.965825326 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512222072907776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16360151030272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32889702 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106855 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2342482998449664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÏÁî'Ý/N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263116848 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141740455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2342482640064000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263106855 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2342482998449664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784353493272062e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16188289007104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2326294351056896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2314324259225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121366400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121366402 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141740455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16163936796936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59691588 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141750446 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2326294368096256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2330488213061896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2330487855472534 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26801886 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚512222054295552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59691311 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚13151.965825326 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512222072907776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16360151030272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32889702 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106855 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2342482998449664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26887391 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843535332907488e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513860063463424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530270661124096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2333509161508864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2349747491769344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2333509178548224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2337716841416698 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2349747491769344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16238313221120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16237971881984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530270551837696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142195832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚1263 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121740370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26887388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121740370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32991872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59878991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321503299186688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16213201680120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16410488374272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚15349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16213542230010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121740368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2349747133390848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142185855 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16237971881984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32991603 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16410579048448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263926223`ó¢âkì/†P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16213542230010 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121740368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2349747133390848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142185855 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16237971881984 TGDeviceStatsMonitor.on_train_bat|„åk¡!ch_start/inactive_split.small_pool.freed‚32991603 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16410579048448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263926223 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2333509178548224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121740368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2349747133390848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59879263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513860082075648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13191.984469968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2337716483827448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2333509161508864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26887391 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843535332907488e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513860063463424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530270661124096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2333509161508864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2349747491769344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2333509178548224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2337716841416698 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2349747491769344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16238313221120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16237971881984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530270551837696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142195832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚1263 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121740370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26887388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121740370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32991872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59878991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321503299186688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16213201680120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16410488374272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚15349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16411523941888 _step‚1264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121748002 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142204898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2333656402538496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26889133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952900 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121748000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142194907 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843535339041526e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2349895745433088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2337864363142404 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121748000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16239325855232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59882848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2333656402538496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530305112860160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952900 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16239325855232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16238984508928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚15349 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚13192.597883817 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513893510834176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32993964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530305016163840 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2321649792819200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142204898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16238984508928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263942907 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59883100 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2333656419577856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2349895387047424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321649809858560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142194907 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2349895745433088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263942907 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121748002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2333656419577856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26889136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513893492221952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0?C'ï/N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16411602025984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59882848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2333656402538496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530305112860160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952900 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16239325855232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16238984508928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚15349 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚13192.597883817 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513893510834176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32993964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530305016163840 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2321649792819200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142204898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16238984508928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263942907 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59883100 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2333656419577856 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2349895387047424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321649809858560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142194907 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2349895745433088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263942907 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121748002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2333656419577856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26889136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513893492221952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16411523941888 _step‚1264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121748002 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142204898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2333656402538496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26889133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952900 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121748000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142194907 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843535339041526e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2349895745433088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2337864363142404 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121748000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16239325855232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2357159880374272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2357159880374272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26974638 % _timestamp‚1.7843535736857798e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2340871230029824 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60070614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33095973 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚15399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16289008722944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33095711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122121970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16461889265664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚531993491553280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515531520002048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142640307 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2340871230029824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264772254 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2345092991497206 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122121968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122121968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2357160238752768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚531993390655488 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2357159880374272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2357159880374272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26974638 % _timestamp‚1.7843535736857798e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2340871230029824 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60070614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33095973 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚15399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16289008722944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33095711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122121970 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16461889265664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚531993491553280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515531520002048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142640307 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2340871230029824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264772254 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2345092991497206 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122121968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122121968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2357160238752768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚531993390655488 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328975326412800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264778959 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2345240155633550 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2357308492416512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16289680010752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16289680010752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16264829220750 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122129602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33097946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2345240513222912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122129600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778959 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142659350 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2357308492416512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532027845597696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚13232.994852472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2328975343452160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2341018471059456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142659350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16462997727744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7843535743010736e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60074075 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264788952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33097692 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122129600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2357308134030848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60074332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16462915449344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16265169770752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122129602 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2341018454020096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26976386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.curkx§UÆrent‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚15399þÛœ…'0N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122129602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33097946 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2345240513222912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122129600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778959 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142659350 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2357308492416512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532027845597696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚13232.994852472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2328975343452160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2341018471059456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142659350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16462997727744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7843535743010736e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60074075 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264788952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33097692 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122129600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2357308134030848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60074332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16462915449344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16265169770752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122129602 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2341018454020096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26976386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚15399 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26976383 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515564930148352 _step‚1268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2341018471059456 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2341018454020096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328975326412800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264778959 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2345240155633550 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2357308492416512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16289680010752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16289680010752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16264829220750 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269ô …缯4‚0/ _wandbruntime‚13232  _runtime‚132320fKðƒ0Œ 1loss/SRqqt2/detection_loss‚0.5718451377399394 3loss/FBt2/assignment_loss‚0.0002913741806801842 %loss/total_loss‚7.305426062157302 0loss/FRt1/assignment_loss‚1.4457671582698821 3loss/SRqqt1/assignment_loss‚0.48675951142565665 3loss/SRqqt2/assignment_loss‚0.42355778316955006 /loss/FRt2/detection_loss‚1.1856858950746558 /loss/FBt1/detection_loss‚0.5550565397793737 trainer/global_step‚15399 3loss/FBt1/assignment_loss‚0.0038006871777101973 /loss/FRt1/detection_loss‚0.5413513272417653 % _timestamp‚1.7843535743042548e+09 _runtime‚13232.997539285 _step‚1269 0loss/FBt2/detection_loss‚0.36810735913475445 epoch‚38 0loss/FRt2/assignment_loss‚0.7796090481965945 1loss/SRqqt1/detection_loss‚0.9435942407667397Ê\Û„0‘ 1loss/SRqqt1/detection_loss‚0.9435942407667397 1loss/SRqqt2/detection_loss‚0.5718451377399394 3loss/FBt2/assignment_loss‚0.0002913741806801842 %loss/total_loss‚7.305426062157302 0loss/FRt1/assignment_loss‚1.4457671582698821 3loss/SRqqt1/assignment_loss‚0.48675951142565665 3loss/SRqqt2/assignment_loss‚0.42355778316955006 /loss/FRt2/detection_loss‚1.1856858950746558 /loss/FBt1/detection_loss‚0.5550565397793737 trainer/global_step‚15399 3loss/FBt1/assignment_loss‚0.0038006871777101973 /loss/FRt1/detection_loss‚0.5413513272417653 % _timestamp‚1.7843535743042548e+09 _runtime‚13232.997539285 _step‚1269 0loss/FBt2/detection_loss‚0.36810735913475445 epoch‚38 0loss/FRt2/assignment_loss‚0.7796090481965945õ Õ““…0‚(:ˆ ²¦ìÒã盓 gpu.0.gpu‚96.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P+gé†0‚(:Þ ²¦ìÒ﯊— disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 431.15625 disk.rbd1.out‚ 43.0361328125' disk.nvme0c0n1.out‚31581.7294921875 disk.dm-0.in‚ 21604.4921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2823.0625! disk.nvme0n1.in‚ 21604.4921875 network.recv‚ 3.537278e+06 disk.sda.out‚ 431.15625 disk.sda2.in‚0 disk.dm-1.out‚ 431.15625 proc.cpu.threads‚17 network.sent‚ 2.4534009e+07# disk./.usageGB‚585.412769317627 disk.sda.in‚ 0.1328125# disk.nvme0c0n1.in‚ 21604.4921875* proc.memory.percent‚17.230606079101562) disk./.usagePercent‚39.28754214293625" disk.dm-0.out‚31581.7294921875 disk.sda1.out‚0( proc.memory.availableMB‚ 393.78515625 cpu‚26.200533768113395% disk.nvme0n1.out‚31581.7294921875 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.59652614593506 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0»Ü¨)‘‡0‚(:† Á¦ìÒ¶÷Ô” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.653' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úiXéˆ0‚(:Þ Á¦ìÒÈìÒ• disk.dm-1.out‚431.4375( disk.nvme0c0n1.out‚31585.53955078125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 21604.4921875 disk.dm-0.in‚ 21604.4921875$ memory_percent‚97.59323596954346 disk.rbd1.out‚ 43.0361328125 disk.sda1.out‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2823.0625 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 21604.4921875* proc.memory.percent‚17.230606079101562 cpu‚26.200764815779486 network.recv‚ 3.541811e+06 disk.sda1.in‚0 network.sent‚ 2.4597635e+07 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 394.32421875) disk./.usagePercent‚39.28761254500817# disk./.usageGB‚585.413818359375 disk.rbd0.out‚0 disk.sda.out‚431.4375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚431.4375 disk.rbd0.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚31585.53955078125& disk.nvme0n1.out‚31585.53955078125 disk.rbd1.in‚ 2.01953125 proc.cpu.threads‚17 “Ü4‰0/ _wandbruntime‚13273  _runtime‚13273[ÍÉàÊŠ0Ä & lr-AdamW/pg2‚5.713710358158501e-05 trainer/global_step‚15449 % _timestamp‚1.7843536143707566e+09 _runtime‚13273.064348375 _step‚1270 & lr-AdamW/pg1‚5.713710358158501e-05ÊÇþÏÏ‹0É trainer/global_step‚15449 % _timestamp‚1.7843536143707566e+09 _runtime‚13273.064348375 _step‚1270 & lr-AdamW/pg1‚5.713710358158501e-05 & lr-AdamW/pg2‚5.713710358158501e-05ö ý¢ûI4Œ0/ _wandbruntime‚13273  _runtime‚13273tjc¤(0ÿO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16339362885632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13273.06557508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚15449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143094759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2352468783988464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16314434653936 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16339362885632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122503568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16339704224768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608306 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60261401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16513254079488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2352469141577714 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2364572627357696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143104736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843536143718483e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27061891 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16314775203826 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2348233281511424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533716292196352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2364572985736192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143104736 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2364572627357696 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2364572985736192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533716193395712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚517202957928448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2348233264472064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16513334267904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265598327 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60261655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122503568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33199764 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33199513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60261655 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122503568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33199764 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33199513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143094759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2352468783988464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16314434653936 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16339362885632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122503568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16339704224768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608306 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60261401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16513254079488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2352469141577714 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2364572627357696 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143104736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843536143718483e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27061891 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16314775203826 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2348233281511424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533716292196352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2364572985736192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143104736 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2364572627357696 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2364572985736192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533716193395712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚517202957928448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2348233264472064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16513334267904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0÷ °ÉÌ40/ _wandbruntime‚13273  _runtime‚13273õIc'0ŠN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265625004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33201548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2352616305714058 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2336300860006400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2364720881014272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143103811 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2352616663303420 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2348380522541056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315445707658 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122511200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143113802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16340716858880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265615011 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27063633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533750658549248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16315786257660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚15449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16340375512576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336300877045760 _runtime‚13273.681457309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122511202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33201792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122511200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2348380522541056 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16340716858880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265625004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60265181 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚01úÈ'‘0N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265615011 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16514368558592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843536149876504e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60265428 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122511202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33201792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122511200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2348380522541056 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16340716858880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265625004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60265181 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265625004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33201548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2352616305714058 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2336300860006400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2364720881014272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143103811 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2352616663303420 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2348380522541056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315445707658 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122511200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143113802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16340716858880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265615011 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27063633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533750658549248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16315786257660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚15449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16340375512576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336300877045760 _runtime‚13273.681457309 Eù4›0/ _wandbruntime‚13313  _runtime‚13313Ì)9µÆœ0À $ lr-AdamW/pg1‚5.5943179137527e-05 $ lr-AdamW/pg2‚5.5943179137527e-05 trainer/global_step‚15499 % _timestamp‚1.7843536548367512e+09 _runtime‚13313.530330682 _step‚1274Õ?ÅôË0Å % _timestamp‚1.7843536548367512e+09 _runtime‚13313.530330682 _step‚1274 $ lr-AdamW/pg1‚5.5943179137527e-05 $ lr-AdamW/pg2‚5.5943179137527e-05 trainer/global_step‚15499ú „?­4ž0/ _wandbruntime‚13313  _runtime‚13313X©}v› Ÿ0P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266434379 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266444358 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143549211 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60452566 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2359844934068972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16365391690734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343479882928128 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2355595315953664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16564744254976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16365051140844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535439039212032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143559188 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2371985374341120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122885168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143559188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMoni%0¶ltor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2359845291658222 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122885170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16390399726592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2371985732719616 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122885168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16390058387456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518874377242624 _step‚1275 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2355595332993024 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2355595315953664 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚15499 _runtime‚13313.531613181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16390058387456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60452820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27149138 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16564661969408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122885170 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33303428 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 íwÑ ( 0†P _runtime‚13313.531613181 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16390058387456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60452820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27149138 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16564661969408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122885170 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33303428 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266434379 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266444358 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143549211 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60452566 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2359844934068972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16365391690734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343479882928128 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2355595315953664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16564744254976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16365051140844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535439039212032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143559188 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2371985374341120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122885168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143559188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2359845291658222 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122885170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16390399726592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2371985732719616 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122885168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16390058387456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518874377242624 _step‚1275 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2355595332993024 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2355595315953664 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚15499û •}ï˜4¡0/ _wandbruntime‚13314  _runtime‚13314^¾p'¢0ŠN KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122892802 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2355742574022656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16391071014400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2355742556983296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2355742574022656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16391412360704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16366402744568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27150883 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16366062194566 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122892802 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843536554513676e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2372133627997696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2343626410639360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122892800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122892800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16391071014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚13314.145103033 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266461056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143568254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33305474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60456357 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535473602094080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27150886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33305707 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2359992813383928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266451063 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1276 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16565701493760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60456593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143568254 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2372133986383360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2355742556983296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2359992455794566<‘‘p!£0N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518907824613376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚518907806001152 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1276 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16565701493760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60456593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143568254 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2372133986383360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2355742556983296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2359992455794566 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122892802 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2355742574022656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16391071014400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2355742556983296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2355742574022656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16391412360704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_splitOQéäô.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16366402744568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27150883 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16366062194566 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122892802 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843536554513676e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2372133627997696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2343626410639360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122892800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122892800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16391071014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚13314.145103033 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266461056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143568254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33305474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60456357 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535473602094080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27150886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33305707 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2359992813383928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266451063 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2367221084149480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚13354.166522876 _step‚1279 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537161906898944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2362957367435264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16616169821184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144003663 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2367221441738730 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16415667627752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚15549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16441095228416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123266768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144003663 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2379398121324544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2379398121324544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2362957367435264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2350805433561088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123266768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33407129 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123266770 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27236388 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2367221084149480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚13354.166522876 _step‚1279 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537161906898944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2362957367435264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16616169821184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144003663 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2367221441738730 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16415667627752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚15549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16441095228416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123266768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144003663 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2379398121324544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2379398121324544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2362957367435264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2350805433561088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123266768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33407129 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123266770 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27236388 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123274402 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144022706 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520579262539776 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2363104608464896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60647579 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 1440127-Þ·\Ä"15 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33409443 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚15549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27238133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2363104625504256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123274400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16441766516224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16416678681474 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843536960870466e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267297108 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60647308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2367368963464436 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2363104608464896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537196463936000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520579243927552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16617201396224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350951927193600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2363104625504256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33409175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚1280 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123274402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123274400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144012715 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287115 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16617100243456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2379546733366784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088¹¢j'µ0N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2379546733366784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144022706 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2379546733366784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123274402 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144022706 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520579262539776 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2363104608464896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60647579 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144012715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33409443 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚15549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27238133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2363104625504256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123274400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16441766516224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16416678681474 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843536960870466e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267297108 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60647308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2367368963464436 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2363104608464896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537196463936000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520579243927552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16617201396224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350951927193600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2363104625504256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33409175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚1280 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123274402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123274400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144012715 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287115 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16617100243456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2€ Púý®4¶0/ _wandbruntime‚13354  _runtime‚13354Ò”«·0ˆ 3loss/FBt2/assignment_loss‚0.0005489695117461129 /loss/FRt2/detection_loss‚1.2112338432663192 /loss/FBt2/detection_loss‚0.4907283915657802 trainer/global_step‚15549 % _timestamp‚1.7843536960900352e+09 _runtime‚13354.783331814 0loss/FRt2/assignment_loss‚0.6484623430059847 0loss/FBt1/detection_loss‚0.42206966419061326 epoch‚38 3loss/SRqqt1/assignment_loss‚0.38589227811238824 .loss/FRt1/detection_loss‚0.538398671364481 1loss/SRqqt1/detection_loss‚0.8419029328331019 %loss/total_loss‚7.070338806394983 0loss/FRt1/assignment_loss‚1.4913473954263903 2loss/SRqqt2/assignment_loss‚0.4509165077986476 1loss/FBt1/assignment_loss‚0.00460724523238909 1loss/SRqqt2/detection_loss‚0.5842305640871412 _step‚1281·°7š¸0 0loss/FRt1/assignment_loss‚1.4913473954263903 2loss/SRqqt2/assignment_loss‚0.4509165077986476 1loss/FBt1/assignment_loss‚0.00460724523238909 1loss/SRqqt2/detection_loss‚0.5842305640871412 _step‚1281 3loss/FBt2/assignment_loss‚0.0005489695117461129 /loss/FRt2/detection_loss‚1.2112338432663192 /loss/FBt2/detection_loss‚0.4907283915657802 trainer/global_step‚15549 % _timestamp‚1.7843536960900352e+09 _runtime‚13354.783331814 0loss/FRt2/assignment_loss‚0.6484623430059847 0loss/FBt1/detection_loss‚0.42206966419061326 epoch‚38 3loss/SRqqt1/assignment_loss‚0.38589227811238824 .loss/FRt1/detection_loss‚0.538398671364481 1loss/SRqqt1/detection_loss‚0.8419029328331019 %loss/total_loss‚7.070338806394983 O™‡¯¹0‚(:’ ª§ìÒâ–ד gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ•÷'ëº0‚(:à ª§ìÒé·÷• disk.sda3.in‚ 0.1328125$ memory_percent‚97.59736061096191* proc.memory.percent‚17.230606079101562 cpu‚26.20223221917056 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0$ disk./.usageGB‚585.4925270080566 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚31714.47119140625 disk.rbd0.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 393.6484375 proc.memory.rssMB‚ 2823.0625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚436.1875 disk.rbd1.in‚ 2.01953125 network.recv‚ 3.567244e+06 disk.dm-1.out‚436.1875$ disk.nvme0c0n1.in‚21791.37109375 disk.dm-0.in‚21791.37109375 network.sent‚ 2.4748228e+07) disk./.usagePercent‚39.29289474846215& disk.nvme0n1.out‚31714.47119140625 disk.sda3.out‚436.1875 disk.rbd1.out‚ 43.3486328125 disk.sda1.out‚0# disk.dm-0.out‚31714.47119140625 disk.sda2.out‚0 disk.sda.in‚ 0.1328125" disk.nvme0n1.in‚21791.37109375ÞaÇz“»0‚(:ˆ ¹§ìÒ€°à“ gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M†lIí¼0‚(:â ¹§ìÒ‰±– network.sent‚ 2.4811531e+07$ disk./.usageGB‚585.4925155639648 disk.sda3.out‚ 436.421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 393.8671875& disk.nvme0c0n1.out‚31717.029296875 disk.sda1.out‚0 proc.memory.rssMB‚ 2823.0625 proc.cpu.threads‚17 network.recv‚ 3.572209e+06" disk.nvme0n1.in‚21791.37109375$ disk.nvme0n1.out‚31717.029296875 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚21791.37109375 disk.sda.out‚ 436.421875 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 43.3486328125 disk.sda1.in‚0 disk.dm-0.in‚21791.37109375! disk.dm-0.out‚31717.029296875 disk.sda.in‚ 0.1328125$ memory_percent‚97.59602546691895 cpu‚26.202432958499315* disk./.usagePercent‚39.292893980439544 disk.rbd0.out‚0 disk.dm-1.out‚ 436.421875* proc.memory.percent‚17.230606079101562u‘Ú…4½0/ _wandbruntime‚13394  _runtime‚13394êl„âȾ0 & lr-AdamW/pg1‚5.358789191916298e-05 & lr-AdamW/pg2‚5.358789191916298e-05 trainer/global_step‚15599 # _timestamp‚1.78435373612767e+09 _runtime‚13394.821260307 _step‚1282`­¯Í¿0Ç # _timestamp‚1.78435373612767e+09 _runtime‚13394.821260307 _step‚1282 & lr-AdamW/pg1‚5.358789191916298e-05 & lr-AdamW/pg2‚5.358789191916298e-05 trainer/global_step‚15599‚ &s_4À0/ _wandbruntime‚13394  _runtime‚13394 ³Sj"Á0P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123648368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144458115 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16491449391104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16667461686784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2386810868307968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2358130967154688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123648370 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538884733394432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16466624664550 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16491790730240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16491790730240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2386811226686464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144458115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚15599 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370319418916864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522217253095424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1283 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27323641 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123648368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2386811226686464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16466284114660 JDevicIgÑeStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2386810868307968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2370319435956224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60833865 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2358130967154688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123648370 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538884733394432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16466624664550 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16491790730240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16491790730240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2386811226686464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144458115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚15599 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370319418916864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522217253095424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1283 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27323641 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123648368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2386811226686464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16466284114660 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2386810868307968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2370319435956224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60833865 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60837345 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16668489960448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2386959121964544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123656002 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144467167 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2358277477826560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16492462018048 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2386959480350208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16492803364352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2386959480350208 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2370466676985856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123656000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144477158 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144467167 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268133160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538919089536000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2374745113544944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2370466659946496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2370466659946496 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33511962 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚15599 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16492462018048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27325386 % _timestamp‚1.7843537367439065e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144477158 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60837602 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2374744755955582 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16492803364352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27325383 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983ù‡4FP%Ç0N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚15599 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16492462018048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27325386 % _timestamp‚1.7843537367439065e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144477158 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60837602 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2374744755955582 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16492803364352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27325383 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60837345 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16668489960448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2386959121964544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123656002 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144467167 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2358277477826560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16492462018048 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2386959480350208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16492803364352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2386959480350208 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2370466676985856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123656000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144477158 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144467167 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268133160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538919089536000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2374745113544944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2370466659946496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2370466659946496 LCDeviceStatsMonitor.on_tB3ZÆrain_batch_end/allocation.small_pool.current‚9991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33511962„ ÁìP4È0/ _wandbruntime‚13395  _runtime‚13395mSµ É0‹ %loss/total_loss‚6.946540041833323 _step‚1285 0loss/FBt2/detection_loss‚0.26187169068297783 0loss/FRt2/assignment_loss‚0.8021415767031316 /loss/FRt1/detection_loss‚0.5560902577615636 1loss/SRqqt1/detection_loss‚0.8054798583213526 trainer/global_step‚15599 2loss/FBt2/assignment_loss‚0.002213535544772954 epoch‚38 0loss/FBt1/detection_loss‚0.49008065186373295 0loss/FRt1/assignment_loss‚1.4324042242321362 3loss/SRqqt1/assignment_loss‚0.36499899144065207 2loss/SRqqt2/assignment_loss‚0.4504714020040057 /loss/FRt2/detection_loss‚1.1999059835726675 1loss/SRqqt2/detection_loss‚0.5808526238080953 $ _timestamp‚1.784353736746795e+09 _runtime‚13395.440094661 4loss/FBt1/assignment_loss‚2.9245898235440047e-05Taÿ8Ê0 0loss/FBt2/detection_loss‚0.26187169068297783 0loss/FRt2/assignment_loss‚0.8021415767031316 /loss/FRt1/detection_loss‚0.5560902577615636 1loss/SRqqt1/detection_loss‚0.8054798583213526 trainer/global_step‚15599 2loss/FBt2/assignment_loss‚0.002213535544772954 epoch‚38 0loss/FBt1/detection_loss‚0.49008065186373295 0loss/FRt1/assignment_loss‚1.4324042242321362 3loss/SRqqt1/assignment_loss‚0.36499899144065207 2loss/SRqqt2/assignment_loss‚0.4504714020040057 /loss/FRt2/detection_loss‚1.1999059835726675 1loss/SRqqt2/detection_loss‚0.5808526238080953 $ _timestamp‚1.784353736746795e+09 _runtime‚13395.440094661 4loss/FBt1/assignment_loss‚2.9245898235440047e-05 %loss/total_loss‚6.946540041833323 _step‚1285… 42S‘Ë0‚(:† קìÒ¼ãÜ” gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“CáàöÌ0‚(:ë קìÒ·‰Ò• disk.sda1.out‚0 disk.dm-1.out‚ 437.046875 cpu‚26.206448820694714( disk.nvme0c0n1.out‚31725.11767578125* proc.memory.percent‚17.238640785217285# disk./.usageGB‚585.492805480957$ memory_percent‚97.61583805084229 network.recv‚ 3.580543e+06 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 43.4736328125 disk.rbd0.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2824.37890625 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚31725.11767578125 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚21791.37109375 disk.dm-0.in‚21791.37109375( proc.memory.availableMB‚ 390.62109375 disk.sda3.out‚ 437.046875 disk.drbd2165.in‚0 network.sent‚ 2.4868932e+07 disk.sda.out‚ 437.046875 disk.sda1.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚31725.11767578125 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚21791.37109375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125) disk./.usagePercent‚39.29291343701215L¥ù4Í0/ _wandbruntime‚13413  _runtime‚13413»Ï$ ¡=Î0›{ >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843537546926718e+09 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2616029679775238 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31792786717414856 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6741927266120911 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035641103982925415 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.8995311260223389 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6939177513122559 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5034176111221313 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6558218002319336 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23611262440681458 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36505240201950073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947980046272278 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04319088160991669 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22646118700504303 &particle/f_score‚0.605350911617279 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33240050077438354 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9447077512741089 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2239610105752945 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09443360567092896 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24409544467926025 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1261141151189804 +jet/accuracy_3_of_4‚0.04078979045152664 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13139621913433075 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5742248892784119 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03274478763341904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4996044933795929 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6721532940864563 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024542924016714096 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6400482058525085 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37780317664146423 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02434670366346836 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8369237184524536 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712462544441223 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6542506217956543 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.637784481048584 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961495995521545 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7839001417160034 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04891834780573845 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30873560905456543 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.659430205821991 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2112625688314438 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7916048765182495 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20331664383411407 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2280561774969101 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.718596875667572 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19780580699443817 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7269991636276245 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚13413.386855318 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0270750243216753 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34171944856643677 _step‚1286 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026771631091833115 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647601366043091 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5271855592727661 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6315914392471313 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7047852277755737 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_2‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7847906947135925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876638054847717 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚38 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27002063393592834 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2372620701789856 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7628161907196045 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15567204356193542 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7928982973098755 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7152066230773926 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8814513683319092 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8318405151367188 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_4‚0.9806751012802124 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13839027285575867 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.615367591381073 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09700775146484375 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7191764712333679 *jet/accuracy_2_of_4‚0.7647850513458252 5validation_average_jet_accuracy‚0.375764399766922 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7805933356285095 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7892848253250122 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.037970054894685745 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676832795143127 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6296769380569458 *jet/accuracy_1_of_3‚0.9252341985702515 particle/accuracy_1_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.640410304069519 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6459908485412598 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6400482058525085 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.676276445388794 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23713809251785278 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14066332578659058 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01738065667450428 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22552083432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12363412231206894 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01738065667450428 *jet/accuracy_2_of_3‚0.5063894987106323 +particle/sensitivity‚0.5935183167457581 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957664012908936 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6736367344856262 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8814513683319092 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9968254566192627 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.994434654712677 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7346700429916382 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26962074637413025 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988112211227417 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7357820272445679 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2804892361164093 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26962074637413025 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.713503897190094 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20295913517475128 )loss/val_total_loss‚7.655921247208591 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953811168670654 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_4_of_5‚0 trainer/global_step‚15599 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704837560653687 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2417563498020172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5454166531562805 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2821873724460602 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08318576961755753 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6954559683799744 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26334720849990845 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7349370121955872 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9432940483093262 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03547905385494232 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807032704353333 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 particle/accuracy_2_of_2‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1436121165752411 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8338139653205872 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6"Purity/0FRt*SRqqt1FBt/event_purity‚0.7681924700737 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7200809717178345 *jet/accuracy_1_of_2‚0.7845686078071594 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5427994728088379 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5712005496025085 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26914066076278687 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28064513206481934 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 úåD4Ï0 { >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_2‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7847906947135925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876638054847717 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚38 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27002063393592834 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2372620701789856 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7628161907196045 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15567204356193542 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7928982973098755 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7152066230773926 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8814513683319092 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8318405151367188 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_4‚0.9806751012802124 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13839027285575867 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.615367591381073 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09700775146484375 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7191764712333679 *jet/accuracy_2_of_4‚0.7647850513458252 5validation_average_jet_accuracy‚0.375764399766922 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7805933356285095 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7892848253250122 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.037970054894685745 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676832795143127 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6296769380569458 *jet/accuracy_1_of_3‚0.9252341985702515 particle/accuracy_1_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.640410304069519 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6459908485412598 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6400482058525085 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.676276445388794 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23713809251785278 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14066332578659058 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01738065667450428 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22552083432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12363412231206894 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01738065667450428 *jet/accuracy_2_of_3‚0.5063894987106323 +particle/sensitivity‚0.5935183167457581 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957664012908936 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6736367344856262 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8814513683319092 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9968254566192627 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.994434654712677 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7346700429916382 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26962074637413025 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988112211227417 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7357820272445679 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2804892361164093 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26962074637413025 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.713503897190094 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20295913517475128 )loss/val_total_loss‚7.655921247208591 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953811168670654 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_4_of_5‚0 trainer/global_step‚15599 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704837560653687 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2417563498020172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5454166531562805 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2821873724460602 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08318576961755753 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6954559683799744 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26334720849990845 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7349370121955872 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9432940483093262 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03547905385494232 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807032704353333 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 particle/accuracy_2_of_2‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1436121165752411 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8338139653205872 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6"Purity/0FRt*SRqqt1FBt/event_purity‚0.7681924700737 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7200809717178345 *jet/accuracy_1_of_2‚0.7845686078071594 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5427994728088379 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5712005496025085 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26914066076278687 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28064513206481934 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843537546926718e+09 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2616029679775238 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31792786717414856 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6741927266120911 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035641103982925415 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.8995311260223389 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6939177513122559 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5034176111221313 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6558218002319336 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23611262440681458 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36505240201950073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947980046272278 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04319088160991669 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22646118700504303 &particle/f_score‚0.605350911617279 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33240050077438354 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9447077512741089 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2239610105752945 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09443360567092896 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24409544467926025 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1261141151189804 +jet/accuracy_3_of_4‚0.04078979045152664 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13139621913433075 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5742248892784119 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03274478763341904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4996044933795929 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6721532940864563 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024542924016714096 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6400482058525085 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37780317664146423 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02434670366346836 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8369237184524536 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712462544441223 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6542506217956543 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.637784481048584 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961495995521545 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7839001417160034 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04891834780573845 ("Purity/2FRt*SRqqt2FBtøK£b /SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30873560905456543 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.659430205821991 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2112625688314438 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7916048765182495 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20331664383411407 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2280561774969101 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.718596875667572 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19780580699443817 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7269991636276245 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚13413.386855318 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0270750243216753 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.34171944856643677 _step‚1286 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026771631091833115 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647601366043091 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5271855592727661 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6315914392471313 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7047852277755737† &õñIœÐ0‚(:‘ æ§ìҌī“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÕ½úÑ0‚(:ï æ§ìÒ—Ÿ¼–$ memory_percent‚97.63345718383789 cpu‚26.207497420171908 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 43.5986328125# proc.memory.rssMB‚ 2827.01953125* disk./.usagePercent‚39.292913949027216 disk.dm-1.out‚ 437.37109375# disk.dm-0.out‚31732.16259765625& disk.nvme0n1.out‚31732.16259765625 disk.dm-0.in‚21864.63671875 disk.sda.out‚ 437.37109375 disk.sda2.in‚0" disk.nvme0n1.in‚21864.63671875 disk.rbd1.in‚ 2.01953125( disk.nvme0c0n1.out‚31732.16259765625 network.recv‚ 3.584804e+06 disk.sda1.out‚0 disk.sda3.out‚ 437.37109375 network.sent‚ 2.492152e+07 disk.drbd2165.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.25475788116455 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚21864.63671875 disk.sda3.in‚ 0.1328125& proc.memory.availableMB‚ 387.734375$ disk./.usageGB‚585.4928131103516 disk.sda1.in‚0 disk.sda.in‚ 0.1328125t·“Ò0‚(:ˆ õ§ìÒÄñÈ• gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07 óéÓ0‚(:Þ õ§ìÒòð–$ disk./.usageGB‚585.4927520751953 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21864.63671875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125) proc.memory.percent‚17.25475788116455 disk.rbd1.in‚ 2.01953125( disk.nvme0c0n1.out‚31735.14794921875 disk.drbd2165.out‚0# disk.dm-0.out‚31735.14794921875 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚31735.14794921875 disk.rbd0.out‚0 disk.sda3.in‚ 0.1328125) disk./.usagePercent‚39.29290985290667 disk.dm-1.out‚437.75 disk.sda.out‚437.75 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚437.75( proc.memory.availableMB‚ 388.17578125 disk.drbd2165.in‚0" disk.nvme0n1.in‚21864.63671875 proc.cpu.threads‚17$ memory_percent‚97.63076305389404 network.recv‚ 3.589265e+06 disk.sda1.in‚0 disk.dm-0.in‚21864.63671875# proc.memory.rssMB‚ 2827.01953125 network.sent‚ 2.4981073e+07 disk.rbd1.out‚ 43.5986328125 cpu‚26.20769158287684ý‚¼4Ô0/ _wandbruntime‚13453  _runtime‚13453˜Îƒ­ÊÕ0Ä _step‚1287 & lr-AdamW/pg1‚5.242668042055396e-05 & lr-AdamW/pg2‚5.242668042055396e-05 trainer/global_step‚15649 % _timestamp‚1.7843537947478075e+09 _runtime‚13453.441457474h¤teÏÖ0É & lr-AdamW/pg1‚5.242668042055396e-05 & lr-AdamW/pg2‚5.242668042055396e-05 trainer/global_step‚15649 % _timestamp‚1.7843537947478075e+09 _runtime‚13453.441457474 _step‚1287‡ Žœ4×0/ _wandbruntime‚13453  _runtime‚13453A?(Ø0€P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33645481 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144982889 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16559878098432 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124150634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61059152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16560219437568 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚13453.445422557 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2396170915147264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383900124839055 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2379611054088192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16560219437568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61058889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523904621733888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16534620326797 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2396171273525760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379611037048832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144982889 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2379611037048832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269133523 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144992866 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚540641314369024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚15649 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2396170915147264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16559878098432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144992866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33645741 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2367365146923008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143502 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2396171273525760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚540641413169664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534960876687åP‚ (Ù0…P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144982889 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16559878098432 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124150634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61059152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16560219437568 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚13453.445422557 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2396170915147264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383900124839055 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2379611054088192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16560219437568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61058889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523904621733888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16534620326797 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2396171273525760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379611037048832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144982889 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2379611037048832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269133523 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144992866 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚540641314369024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚15649 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2396170915147264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16559878098432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144992866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33645741 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2367365146923008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143502 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2396171273525760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚540641413169664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534960876687 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33645481 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544ˆ r1hØ4Ú0/ _wandbruntime‚13454  _runtime‚13454*‡¬ëÛ0‹N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124158266 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2379758278078464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61062697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33647544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16737723512832 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2379758295117824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269150207 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2396319527189504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2384047646564761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33647815 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15649 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145001932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2367511674634240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144991941 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16560890725376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843537953635921e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144991941 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269160200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269150207 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2379758295117824 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2396319168803840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27415156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523938031880192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27415153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540675864672256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61062971 _step‚1289 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2379758278078464 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16561232071680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535971930521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Ónw['Ü0N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269150207 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2396319527189504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2384047646564761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33647815 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15649 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145001932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2367511674634240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144991941 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16560890725376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843537953635921e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144991941 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269160200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269150207 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2379758295117824 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2396319168803840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27415156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523938031880192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27415153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540675864672256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61062971 _step‚1289 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2379758278078464 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16561232071680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535971930521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124158266 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2379758278078464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61062697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33647544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16737723512832 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2379758295117824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888‰ HáCÃ4Ý0/ _wandbruntime‚13454  _runtime‚13454»è"Þ0Š 2loss/SRqqt2/assignment_loss‚0.3906923768937013 .loss/FRt2/detection_loss‚1.233240217897005 1loss/SRqqt1/detection_loss‚0.9555044384305349 /loss/FRt1/assignment_loss‚1.433450805745237 3loss/FBt1/assignment_loss‚3.143316458745148e-05 2loss/FBt2/assignment_loss‚0.001495338415224879 2loss/SRqqt2/detection_loss‚0.48221304680281274 0loss/FBt2/detection_loss‚0.42269038003052206 /loss/FRt1/detection_loss‚0.5782012564775955 0loss/FBt1/detection_loss‚0.44583032061005146 %loss/total_loss‚6.953435898983967 epoch‚39 _runtime‚13454.060739827 _step‚1290 0loss/FRt2/assignment_loss‚0.6323771586447405 3loss/SRqqt1/assignment_loss‚0.37770912587195365 trainer/global_step‚15649 % _timestamp‚1.7843537953674238e+09÷²[ß0 % _timestamp‚1.7843537953674238e+09 2loss/SRqqt2/assignment_loss‚0.3906923768937013 .loss/FRt2/detection_loss‚1.233240217897005 1loss/SRqqt1/detection_loss‚0.9555044384305349 /loss/FRt1/assignment_loss‚1.433450805745237 3loss/FBt1/assignment_loss‚3.143316458745148e-05 2loss/FBt2/assignment_loss‚0.001495338415224879 2loss/SRqqt2/detection_loss‚0.48221304680281274 0loss/FBt2/detection_loss‚0.42269038003052206 /loss/FRt1/detection_loss‚0.5782012564775955 0loss/FBt1/detection_loss‚0.44583032061005146 %loss/total_loss‚6.953435898983967 epoch‚39 _runtime‚13454.060739827 _step‚1290 0loss/FRt2/assignment_loss‚0.6323771586447405 3loss/SRqqt1/assignment_loss‚0.37770912587195365 trainer/global_step‚15649Š m¯Ò¬’à0‚(:‡ „¨ìÒᶂ– gpu.0.gpu‚22.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šõ¯Åöá0‚(:ë „¨ìÒ¤æÈ— disk.rbd0.out‚0$ disk.nvme0c0n1.in‚21864.63671875 disk.sda.in‚ 0.1328125 disk.sda2.out‚0 disk.dm-1.out‚ 438.15234375 disk.drbd2165.out‚0 disk.sda.out‚ 438.15234375 disk.sda3.out‚ 438.15234375( proc.memory.availableMB‚ 387.70703125# proc.memory.rssMB‚ 2827.01953125 proc.cpu.threads‚17$ memory_percent‚97.63362407684326& disk.nvme0c0n1.out‚31740.392578125 disk.sda1.in‚0" disk.nvme0n1.in‚21864.63671875 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 43.6611328125 disk.dm-0.in‚21864.63671875 disk.sda2.in‚0) proc.memory.percent‚17.25475788116455 cpu‚26.20781005224467 network.sent‚ 2.4996899e+07 network.recv‚ 3.592361e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125! disk.dm-0.out‚31740.392578125 disk.drbd2165.in‚0$ disk./.usageGB‚585.4928894042969$ disk.nvme0n1.out‚31740.392578125* disk./.usagePercent‚39.292919069177906 disk.rbd0.in‚0}ê ²“â0‚(:ˆ “¨ìÒ¸¨˜” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÖÄ÷ã0‚(:ì “¨ìÒùçô– disk.sda2.in‚0 cpu‚26.2079892252029 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 440.41015625 disk.sda3.out‚ 440.41015625 disk.dm-0.in‚21864.63671875 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 43.6611328125# proc.memory.rssMB‚ 2827.01953125) disk./.usagePercent‚39.30140904703592 disk.dm-1.in‚ 0.1328125% disk.nvme0n1.out‚31773.4873046875$ memory_percent‚97.63333797454834 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚21864.63671875' disk.nvme0c0n1.out‚31773.4873046875 disk.sda2.out‚0$ disk./.usageGB‚585.6193962097168 disk.sda1.out‚0 disk.sda.out‚ 440.41015625" disk.nvme0n1.in‚21864.63671875( proc.memory.availableMB‚ 387.75390625 network.recv‚ 3.596882e+06" disk.dm-0.out‚31773.4873046875 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 network.sent‚ 2.5047108e+07 disk.rbd0.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.254757881164559¾“ä0‚(:ˆ ¢¨ìÒÂØž• gpu.0.gpu‚61.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚109.474' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+`©öíå0‚(:â ¢¨ìÒ˜—Þ•# disk.nvme0n1.out‚31797.29296875$ disk./.usageGB‚585.6160011291504 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 441.5859375 cpu‚26.208188587828882 proc.cpu.threads‚17 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0" disk.nvme0n1.in‚21893.98046875 disk.rbd1.out‚ 43.7236328125) disk./.usagePercent‚39.30118120033044 disk.dm-0.in‚21893.98046875 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 441.5859375 disk.drbd2165.in‚0& proc.memory.availableMB‚ 388.140625 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21893.83984375 network.recv‚ 3.600307e+06 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2827.01953125 disk.dm-0.out‚31797.29296875) proc.memory.percent‚17.25475788116455 disk.rbd0.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 441.5859375 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.63097763061523 network.sent‚ 2.505224e+07% disk.nvme0c0n1.out‚31797.29296875››û!4æ0/ _wandbruntime‚13493  _runtime‚13493]Ì{^Ìç0Æ ' lr-AdamW/pg1‚5.1276523340657846e-05 ' lr-AdamW/pg2‚5.1276523340657846e-05 trainer/global_step‚15699 % _timestamp‚1.7843538351687415e+09 _runtime‚13493.862310775 _step‚1291 þÿÑè0Ë ' lr-AdamW/pg1‚5.1276523340657846e-05 ' lr-AdamW/pg2‚5.1276523340657846e-05 trainer/global_step‚15699 % _timestamp‚1.7843538351687415e+09 _runtime‚13493.862310775 _step‚1291‹ |È’Ý4é0/ _wandbruntime‚13493  _runtime‚134931((ê0€P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2403583662130688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16585577363595 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16585236813705 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843538351698027e+09 _step‚1292 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124532234 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61249292 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16610573600256 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386973105569792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13493.86352636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145437341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374690697555968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16610914939392 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2386973088530432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145447318 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27500658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374690680516608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16787997856256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525576041048064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2403584020509184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269979554 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145447318 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2403583662130688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61249569 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525576059660288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124532236 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269969575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2391275917330313 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2403584020509184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2403583662130688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16585577363595 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16585236813705 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843538351698027e+09 ‚I´_step‚1292 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124532234 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61249292 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16610573600256 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386973105569792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13493.86352636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145437341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374690697555968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16610914939392 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2386973088530432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145447318 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27500658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374690680516608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16787997856256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525576041048064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2403584020509184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269979554 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145447318 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2403583662130688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61249569 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525576059660288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Œ Úê6~4ì0/ _wandbruntime‚13494  _runtime‚134948à8y'í0‹N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525609488418816 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2374837208227840 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387120329560064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 _step‚1293 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124539868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269986259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16789016400896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27502406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145456384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚542398593389568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2403732274172928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16611927573504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61253092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33750689 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 % _timestamp‚1.7843538357837899e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145446393iëŸx'î0N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚542398593389568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2403732274172928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16611927573504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61253092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33750689 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 % _timestamp‚1.7843538357837899e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145446393 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525609488418816 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2374837208227840 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387120329560064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 _step‚1293 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124539868 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269986259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16789016400896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27502406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145456384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 .øû4ï0/ _wandbruntime‚13494  _runtime‚13494”&¨ð0ˆ 0loss/FRt1/assignment_loss‚1.4139301643014592 3loss/SRqqt1/assignment_loss‚0.39795830626780604 1loss/SRqqt1/detection_loss‚0.8933649735927984 /loss/FBt1/detection_loss‚0.4893190116231698 trainer/global_step‚15699 % _timestamp‚1.7843538357874591e+09 /loss/FRt2/detection_loss‚1.1994829407972598 /loss/FBt2/detection_loss‚0.3570586251617277 $loss/total_loss‚7.02854693118019 _step‚1294 /loss/FRt2/assignment_loss‚0.695091225206852 _runtime‚13494.481356542 epoch‚39 2loss/SRqqt2/assignment_loss‚0.4323313470369841 3loss/FBt1/assignment_loss‚0.0008830098559778717 3loss/FBt2/assignment_loss‚0.0037484613668628325 /loss/FRt1/detection_loss‚0.5925961693782167 1loss/SRqqt2/detection_loss‚0.5527826965910758Wîi‡ñ0 2loss/SRqqt2/assignment_loss‚0.4323313470369841 3loss/FBt1/assignment_loss‚0.0008830098559778717 3loss/FBt2/assignment_loss‚0.0037484613668628325 /loss/FRt1/detection_loss‚0.5925961693782167 1loss/SRqqt2/detection_loss‚0.5527826965910758 0loss/FRt1/assignment_loss‚1.4139301643014592 3loss/SRqqt1/assignment_loss‚0.39795830626780604 1loss/SRqqt1/detection_loss‚0.8933649735927984 /loss/FBt1/detection_loss‚0.4893190116231698 trainer/global_step‚15699 % _timestamp‚1.7843538357874591e+09 /loss/FRt2/detection_loss‚1.1994829407972598 /loss/FBt2/detection_loss‚0.3570586251617277 $loss/total_loss‚7.02854693118019 _step‚1294 /loss/FRt2/assignment_loss‚0.695091225206852 _runtime‚13494.481356542 epoch‚39Ž ¾X.íò0‚(:â ±¨ìÒ¬¹Ã•" disk.dm-0.out‚31849.7099609375 disk.sda3.in‚ 0.1328125) proc.memory.percent‚17.25475788116455 network.sent‚ 2.5057219e+07 disk.dm-1.out‚ 442.0859375" disk.nvme0c0n1.in‚ 21942.359375' disk./.usagePercent‚39.298175927886 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚31849.7099609375 disk.dm-0.in‚ 21942.359375$ memory_percent‚97.63247966766357 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 442.0859375 cpu‚26.20830977529474 disk.sda2.in‚0 disk.nvme0n1.in‚ 21942.359375 disk.rbd1.in‚ 2.01953125( proc.memory.availableMB‚ 387.89453125 disk.sda1.out‚0 disk.rbd1.out‚ 43.7236328125$ disk./.usageGB‚585.5712203979492 disk.sda1.in‚0% disk.nvme0n1.out‚31849.7099609375 disk.sda3.out‚ 442.0859375# proc.memory.rssMB‚ 2827.01953125 network.recv‚ 3.604022e+06 disk.rbd0.in‚0 disk.rbd0.out‚0yÕ¥žó0‚(:“ ±¨ìÒï’»• gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÂuœžô0‚(:“ À¨ìÒï€ì• gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 »Nìõ0‚(:á À¨ìÒÃñ˜—$ memory_percent‚97.63357639312744) disk./.usagePercent‚39.29762858377779 disk.sda2.out‚0$ disk.nvme0n1.out‚31860.658203125 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 442.4609375) proc.memory.percent‚17.25475788116455 network.sent‚ 2.5122443e+07 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 442.4609375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚26.208570699748673$ disk./.usageGB‚585.5630645751953& disk.nvme0c0n1.out‚31860.658203125 proc.cpu.threads‚17 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125 disk.nvme0n1.in‚ 21942.359375" disk.nvme0c0n1.in‚ 21942.359375 disk.dm-0.in‚ 21942.359375 disk.rbd1.out‚ 43.7861328125 disk.sda.out‚ 442.4609375( proc.memory.availableMB‚ 387.71484375# proc.memory.rssMB‚ 2827.01953125 network.recv‚ 3.60834e+06 disk.sda2.in‚0! disk.dm-0.out‚31860.6582031256\EÁ3ö0‚(:ˆ ϨìÒ™êÖ“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 ½¶œ`gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š†9Hõ÷0‚(:ê ϨìÒëœÅ• disk.nvme0n1.in‚ 21942.359375 network.recv‚ 3.610869e+06( disk./.usagePercent‚39.2976593046819( disk.nvme0c0n1.out‚31865.20947265625 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125) proc.memory.percent‚17.25475788116455$ memory_percent‚97.63209819793701 disk.sda.out‚ 442.80078125 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 21942.359375 disk.sda2.out‚0# proc.memory.rssMB‚ 2827.01953125 cpu‚26.208696923638527 proc.cpu.threads‚17 network.sent‚ 2.5124448e+07 disk.dm-0.in‚ 21942.359375 disk.rbd0.out‚0$ disk./.usageGB‚585.5635223388672 disk.dm-1.in‚ 0.1328125& disk.nvme0n1.out‚31865.20947265625# disk.dm-0.out‚31865.20947265625 disk.sda3.out‚ 442.80078125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 43.7861328125 disk.dm-1.out‚ 442.80078125( proc.memory.availableMB‚ 387.95703125®–4ø0/ _wandbruntime‚13534  _runtime‚13534C Ìù0Æ trainer/global_step‚15749 % _timestamp‚1.7843538757877986e+09 _runtime‚13534.481407998 _step‚1295 ' lr-AdamW/pg1‚5.0137494551914616e-05 ' lr-AdamW/pg2‚5.0137494551914616e-05ì¡GÑú0Ë ' lr-AdamW/pg1‚5.0137494551914616e-05 ' lr-AdamW/pg2‚5.0137494551914616e-05 trainer/global_step‚15749 % _timestamp‚1.7843538757877986e+09 _runtime‚13534.481407998 _step‚1295 ò¼¤‡4û0/ _wandbruntime‚13534  _runtime‚13534#½—(ü0€P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27587908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚527247478974464 _step‚1296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544086741029376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527247497586688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124913836 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16636193850503 $ _timestamp‚1.784353875788938e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145891793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33850677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2394335140012032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124913834 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124913836 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2398652067410821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635853300613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2410996409114112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0³žqÌ (ý0…P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124913836 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2398652067410821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635853300613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2410996409114112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27587908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚527247478974464 _step‚1296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544086741029376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527247497586688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124913836 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16636193850503 $ _timestamp‚1.784353875788938e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145891793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33850677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2394335140012032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124913834 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 §>^;4þ0/ _wandbruntime‚13535  _runtime‚13535·kZª'ÿ0‰N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16662281729024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145900845 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527280907732992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33852924 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16637204904337 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16662281729024 % _timestamp‚1.7843538764028177e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544121300208640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2411145021156352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2394482381041664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16662623075328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2394482398081024 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2411145021156352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16840293682176 trainer/global_step‚15749 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚544121201415168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16636864354335 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145910836 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2394482398081024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61442323 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1297 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124921466 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2398799589136415 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270832304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ŸO¼5€1ŽN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270822311 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33852924 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16637204904337 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16662281729024 % _timestamp‚1.7843538764028177e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544121300208640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2411145021156352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2394482381041664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16662623075328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2394482398081024 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2411145021156352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16840293682176 trainer/global_step‚15749 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚544121201415168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16636864354335 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145910836 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2394482398081024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61442323 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1297 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124921466 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2398799589136415 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270832304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16662281729024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145900845 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527280907732992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146356222 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2401697208532992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16890597635072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125295436 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2401697208532992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61628254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2406028217491329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125295434 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2418409156097536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2401697191493632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125295436 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚15799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979áPŸ†1†P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2401697208532992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16890597635072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125295436 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2401697208532992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61628254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2406028217491329 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125295434 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2418409156097536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2401697191493632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125295436 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚15799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsM å@íonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1300 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16712305943040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146356222” µÐ5“Ž1‚(:ˆ ü¨ìÒ›ùß“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì^Ž9ý1‚(:ò ü¨ìÒ¥àÿ•* disk./.usagePercent‚39.298183608112026" disk.nvme0n1.in‚22059.35546875$ disk.nvme0c0n1.in‚22059.35546875 disk.rbd0.in‚0# disk.dm-0.out‚31878.75537109375( proc.memory.availableMB‚ 387.58203125# proc.memory.rssMB‚ 2827.01953125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 444.25390625 disk.sda2.in‚0 disk.sda3.out‚ 444.25390625 disk.dm-0.in‚22059.35546875) proc.memory.percent‚17.25475788116455 cpu‚26.209236004462657 network.sent‚ 2.5197591e+07 network.recv‚ 3.621499e+06 disk.drbd2165.in‚0$ disk./.usageGB‚585.5713348388672& disk.nvme0n1.out‚31878.75537109375 disk.rbd1.out‚ 43.9736328125 disk.dm-1.out‚ 444.25390625$ memory_percent‚97.63438701629639 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚31878.75537109375<ÕÍÍ41/ _wandbruntime‚13575  _runtime‚13575‡°¸'‘1‹N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16891526951424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16687480841243 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146365288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668356 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2401844432523264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2401844449562624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16712977230848 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚13575.512774777 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125303068 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16891617618432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16713318577152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33955139 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2406175739216923 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389488275415040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545843872610816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16712977230848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2401844432523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389488258375680 % _timestamp‚1.7843539168190222e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33955416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125303066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658363 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146365288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16687821391245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658363 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2418557768139776 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61632042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27676903 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146365288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668356 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2401844432523264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2401844449562624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16712977230848 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚13575.512774777 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125303068 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16891617618432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16713318577152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33955139 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2406175739216923 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389488275415040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545843872610816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16712977230848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2401844432523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389488258375680 % _timestamp‚1.7843539168190222e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33955416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125303066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658363 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146365288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16687821391245 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658363 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2418557768139776 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61632042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27676903 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664• ÈÇÿH4“1/ _wandbruntime‚13575  _runtime‚13575•ú‚”1‹ 0loss/FRt2/assignment_loss‚0.7085060299611559 2loss/SRqqt1/assignment_loss‚0.3623771281886007 1loss/SRqqt2/detection_loss‚0.5450182903213606 _step‚1302 4loss/FBt2/assignment_loss‚4.6187634111374896e-05 /loss/FRt1/detection_loss‚0.5112696919266111 /loss/FRt2/detection_loss‚1.2633810002629544 0loss/FBt1/detection_loss‚0.35591021180955756 epoch‚39 trainer/global_step‚15799 0loss/FRt1/assignment_loss‚1.5338958728350582 3loss/FBt1/assignment_loss‚4.932785011415663e-07 1loss/SRqqt1/detection_loss‚0.8660389909217903 %loss/total_loss‚6.863423490957457 % _timestamp‚1.7843539168215828e+09 2loss/SRqqt2/assignment_loss‚0.4181155282285874 /loss/FBt2/detection_loss‚0.2988640655891679 _runtime‚13575.514884861ÓÛùñ•1 2loss/SRqqt2/assignment_loss‚0.4181155282285874 /loss/FBt2/detection_loss‚0.2988640655891679 _runtime‚13575.514884861 0loss/FRt2/assignment_loss‚0.7085060299611559 2loss/SRqqt1/assignment_loss‚0.3623771281886007 1loss/SRqqt2/detection_loss‚0.5450182903213606 _step‚1302 4loss/FBt2/assignment_loss‚4.6187634111374896e-05 /loss/FRt1/detection_loss‚0.5112696919266111 /loss/FRt2/detection_loss‚1.2633810002629544 0loss/FBt1/detection_loss‚0.35591021180955756 epoch‚39 trainer/global_step‚15799 0loss/FRt1/assignment_loss‚1.5338958728350582 3loss/FBt1/assignment_loss‚4.932785011415663e-07 1loss/SRqqt1/detection_loss‚0.8660389909217903 %loss/total_loss‚6.863423490957457 % _timestamp‚1.7843539168215828e+09– ªÅ©’–1‚(:‡ ‹©ìÒßÈ£” gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖŸ‰ƒú—1‚(:ï ‹©ìÒ篭– disk.rbd1.out‚ 43.9736328125 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚97.63269424438477 proc.cpu.threads‚17 disk.sda3.in‚ 0.1328125 disk.drbd2165.out‚0& proc.memory.availableMB‚ 387.859375 disk.dm-0.in‚22059.35546875 disk.sda2.out‚0( disk.nvme0c0n1.out‚31881.83837890625 network.sent‚ 2.5249845e+07 disk.dm-1.in‚ 0.1328125# disk.dm-0.out‚31881.83837890625$ disk.nvme0c0n1.in‚22059.35546875 disk.sda.out‚ 444.76953125 disk.sda2.in‚0) proc.memory.percent‚17.25475788116455* disk./.usagePercent‚39.298195896473665" disk.nvme0n1.in‚22059.35546875 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.out‚ 444.76953125 network.recv‚ 3.625978e+06$ disk./.usageGB‚585.5715179443359& disk.nvme0n1.out‚31881.83837890625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2827.01953125 cpu‚26.20929783273802 disk.sda3.out‚ 444.76953125 ±gÁž˜1‚(:“ š©ìÒˆ”Ë“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ª|ÿî™1‚(:ã š©ìÒ¢‘ú• disk.drbd2165.out‚0! disk.dm-0.out‚31885.986328125 disk.sda.out‚ 445.328125$ memory_percent‚97.63131141662598 disk.sda1.out‚0$ disk.nvme0n1.out‚31885.986328125 disk.sda3.out‚ 445.328125 disk.drbd2165.in‚0 disk.rbd1.out‚ 44.0361328125) proc.memory.percent‚17.25475788116455* disk./.usagePercent‚39.298196920503806 disk.rbd0.in‚0 disk.dm-1.out‚ 445.328125 disk.dm-0.in‚22059.35546875 disk.sda2.out‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚22059.35546875 disk.sda.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125# proc.memory.rssMB‚ 2827.01953125 network.recv‚ 3.630621e+06# disk./.usageGB‚585.571533203125 disk.sda3.in‚ 0.1328125' proc.memory.availableMB‚ 388.0859375 cpu‚26.20958258278836 disk.sda1.in‚0& disk.nvme0c0n1.out‚31885.986328125 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5302459e+07 disk.dm-1.in‚ 0.1328125" disk.nvme0n1.in‚22059.35546875äÍ‹54š1/ _wandbruntime‚13615  _runtime‚13615z†ˆQÉ›1à $ _timestamp‚1.784353956783732e+09 _runtime‚13615.477371914 _step‚1303 & lr-AdamW/pg1‚4.789311375920147e-05 & lr-AdamW/pg2‚4.789311375920147e-05 trainer/global_step‚158491)sÎœ1È & lr-AdamW/pg2‚4.789311375920147e-05 trainer/global_step‚15849 $ _timestamp‚1.784353956783732e+09 _runtime‚13615.477371914 _step‚1303 & lr-AdamW/pg1‚4.789311375920147e-05— Þ .41/ _wandbruntime‚13615  _runtime‚13615 !Â8ž1P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2396667298336768 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125677034 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425821903080960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚530590354827264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16941665917952 % _timestamp‚1.7843539567848773e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16762660105728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2409059260014592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125677036 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146810674 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16737426824319 _step‚1304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16763001444864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125677036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16737086274429 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2409059260014592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2409059242975232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2425821903080960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27762408 _runtime‚13615.478620031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2409059242975232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146800697 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2413404367571837 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272487710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16941764980736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146810674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547532020745216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16762660105728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16763001444864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61818744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547532138420224ûlP (Ÿ1†P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61818744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547532138420224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2396667298336768 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125677034 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425821903080960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚530590354827264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16941665917952 % _timestamp‚1.7843539567848773e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16762660105728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2409059260014592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125677036 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146810674 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16737426824319 _step‚1304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16763001444864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125677036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16737086274429 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2409059260014592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2409059242975232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2425821903080960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27762408 _runtime‚13615.478620031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2409059242975232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146800697 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2413404367571837 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272487710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16941764980736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146810674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547532020745216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16762660105728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16763001444864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269˜ ~ÎA4 1/ _wandbruntime‚13616  _runtime‚13616÷‚Rô'¡1ŠN SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146819740 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2409206484004864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61822502 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2396813809008640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125684668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34058074 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2425970515123200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2413551889297431 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61822227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16764014078976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272494415 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34058346 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16942782217728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2409206484004864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16763672732672 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16738437878153 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125684666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27764156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27764153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784353957399066e+09 trainer/global_step‚15849 _runtime‚13616.092807206 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2409206501044224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16764014078976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚530623783585792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146809749 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚530623802198016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125684666 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146819740 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547566584415744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0õÿ¡¾¢1N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16738437878153 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125684666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27764156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27764153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784353957399066e+09 trainer/global_step‚15849 _runtime‚13616.092807206 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2409206501044224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16764014078976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚530623783585792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146809749 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚530623802198016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125684666 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146819740 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547566584415744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146819740 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2409206484004864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61822502 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2396813809008640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125684668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34058074 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2425970515123200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2413551889297431 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61822227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16764014078976 ZMDeviceStatsMonitor.on_train_psÏWbatch_end/inactive_split_bytes.small_pool.current‚90667008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272494415 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34058346 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16942782217728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2409206484004864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16763672732672 E°Íø©1‚(:í ¸©ìÒ¢œ–& disk.nvme0n1.out‚31893.42431640625 disk.sda3.in‚ 0.1328125 network.recv‚ 3.638691e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 445.87109375( disk.nvme0c0n1.out‚31893.42431640625( proc.memory.availableMB‚ 388.45703125) proc.memory.percent‚17.25475788116455 cpu‚26.20934092250056 disk.sda2.out‚0 disk.sda2.in‚0$ memory_percent‚97.62904644012451# proc.memory.rssMB‚ 2827.01953125$ disk./.usageGB‚585.5716171264648 disk.drbd2165.out‚0# disk.dm-0.out‚31893.42431640625 disk.sda.out‚ 445.87109375! disk.nvme0n1.in‚ 22152.0234375 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 44.0986328125 disk.sda1.in‚0 disk.sda3.out‚ 445.87109375 proc.cpu.threads‚17 network.sent‚ 2.5370605e+07) disk./.usagePercent‚39.29820255266956 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 22152.0234375 disk.dm-0.in‚ 22152.0234375—9o“ª1‚(:ˆ Ç©ìÒ½ùË“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú³ê«1‚(:ß Ç©ìÒ«ŽÖ• disk.sda.in‚ 0.1328125 disk.sda.out‚446.3125! disk.nvme0n1.in‚ 22152.0234375 network.recv‚ 3.642273e+06$ disk./.usageGB‚585.5715446472168 disk.drbd2165.in‚0 disk.rbd1.out‚ 44.0986328125( disk./.usagePercent‚39.2981976885264 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚446.3125# disk.dm-0.out‚31896.20556640625& disk.nvme0n1.out‚31896.20556640625 cpu‚26.208784802816744 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 22152.0234375 disk.sda2.out‚0 disk.sda3.out‚446.3125 disk.drbd2165.out‚0 disk.sda1.in‚0$ memory_percent‚97.62969017028809) proc.memory.percent‚17.25475788116455( disk.nvme0c0n1.out‚31896.20556640625 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125# proc.memory.rssMB‚ 2827.01953125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0' proc.memory.availableMB‚ 388.3515625 network.sent‚ 2.537563e+07# disk.nvme0c0n1.in‚ 22152.0234375DºOâ4¬1/ _wandbruntime‚13655  _runtime‚13655Rzn^É­1à _step‚1307 & lr-AdamW/pg1‚4.678790590761586e-05 & lr-AdamW/pg2‚4.678790590761586e-05 trainer/global_step‚15899 % _timestamp‚1.7843539969458995e+09 _runtime‚13655.63971535ñ.Øή1È & lr-AdamW/pg1‚4.678790590761586e-05 & lr-AdamW/pg2‚4.678790590761586e-05 trainer/global_step‚15899 % _timestamp‚1.7843539969458995e+09 _runtime‚13655.63971535 _step‚1307› °¨}g4¯1/ _wandbruntime‚13655  _runtime‚13655úð(°1€P TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚15899 _step‚1308 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273323762 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚549254730859520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16787702761337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273313783 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549254825465856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16788043311227 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2420780517652345 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2433235008442880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34159531 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532261811365888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2416421311496192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2433234650064384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27849658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16993014099968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843539969473858e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147265126 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2416421311496192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27849661 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126058634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2403992831930368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚13655.641075268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126058634 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147255149 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2433235008442880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126058636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62009192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147265126 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147255149 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16992938105856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2433234650064384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2420780875241595 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16813696946688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2416421294456832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16813355607552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16813355607552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34159275 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16813355607552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34159275 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2433235008442880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34159531 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532261811365888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2416421311496192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2433234650064384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27849658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16993014099968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843539969473858e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147265126 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2416421311496192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27849661 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126058634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2403992831930368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚13655.641075268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126058634 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147255149 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2433235008442880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126058636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62009192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147265126 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147255149 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16992938105856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2433234650064384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2420780875241595 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16813696946688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3œ ’Èû#4²1/ _wandbruntime‚13656  _runtime‚13656)%à ³1‰N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2416568535486464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2433383262106624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16994044593664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126066268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126066266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2420928039377939 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16814709580800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16993960218112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16788713815059 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2433382903720960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 2ù4ò´ 73340460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147274192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2404139342602240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273330467 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147264201 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34161592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 # _timestamp‚1.78435399756022e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16814368234496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2416568535486464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62012740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2416568552525824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532295221512192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16814368234496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34161337 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15899 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147274192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2404139342602240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273330467 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147264201 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34161592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 # _timestamp‚1.78435399756022e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16814368234496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2416568535486464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62012740 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2416568552525824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532295221512192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16814368234496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34161337 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15899 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273340460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ¨Ð84µ1/ _wandbruntime‚13656  _runtime‚13656;ì§þ¶1ˆ /loss/FRt1/assignment_loss‚1.393152735186413 4loss/FBt1/assignment_loss‚4.8887716273376164e-05 /loss/FRt2/detection_loss‚1.1075604099170235 1loss/SRqqt1/detection_loss‚0.8211651799387409 epoch‚39 _runtime‚13656.257560689 3loss/FBt2/assignment_loss‚0.0008526194258278839 0loss/FBt1/detection_loss‚0.32084165233952394 /loss/FBt2/detection_loss‚0.3553824947528588 2loss/SRqqt2/assignment_loss‚0.4046917207112253 /loss/FRt1/detection_loss‚0.4954785861589936 #loss/total_loss‚6.5190756228308 % _timestamp‚1.7843539975642827e+09 _step‚1310 0loss/FRt2/assignment_loss‚0.6523886012627667 2loss/SRqqt1/assignment_loss‚0.3780046901308557 1loss/SRqqt2/detection_loss‚0.5895080452902979 trainer/global_step‚15899Ä·1 #loss/total_loss‚6.5190756228308 % _timestamp‚1.7843539975642827e+09 _step‚1310 0loss/FRt2/assignment_loss‚0.6523886012627667 2loss/SRqqt1/assignment_loss‚0.3780046901308557 1loss/SRqqt2/detection_loss‚0.5895080452902979 trainer/global_step‚15899 /loss/FRt1/assignment_loss‚1.393152735186413 4loss/FBt1/assignment_loss‚4.8887716273376164e-05 /loss/FRt2/detection_loss‚1.1075604099170235 1loss/SRqqt1/detection_loss‚0.8211651799387409 epoch‚39 _runtime‚13656.257560689 3loss/FBt2/assignment_loss‚0.0008526194258278839 0loss/FBt1/detection_loss‚0.32084165233952394 /loss/FBt2/detection_loss‚0.3553824947528588 2loss/SRqqt2/assignment_loss‚0.4046917207112253 /loss/FRt1/detection_loss‚0.4954785861589936ž ]Ɉ+ž¸1‚(:“ Ö©ìÒŠìá“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ»J‚÷¹1‚(:ì Ö©ìÒ…ò–# disk.nvme0c0n1.in‚ 22152.0234375# proc.memory.rssMB‚ 2827.01953125) proc.memory.percent‚17.25475788116455! disk.nvme0n1.in‚ 22152.0234375 disk.rbd1.out‚ 44.1611328125 disk.sda1.out‚0 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 388.31640625 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚31901.91455078125 disk.sda3.in‚ 0.1328125 network.recv‚ 3.645482e+06 disk.sda1.in‚0 disk.dm-1.out‚ 446.6171875 disk.dm-0.in‚ 22152.0234375 disk.sda3.out‚ 446.6171875 disk.rbd1.in‚ 2.01953125* disk./.usagePercent‚39.298209720880514 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0$ memory_percent‚97.62990474700928 disk.sda.out‚ 446.6171875 disk.sda2.in‚0& disk.nvme0n1.out‚31901.91455078125 proc.cpu.threads‚17 disk.sda2.out‚0 cpu‚26.208260404850833( disk.nvme0c0n1.out‚31901.91455078125 network.sent‚ 2.5380717e+07$ disk./.usageGB‚585.5717239379883 disk.rbd0.in‚0•xŽã“º1‚(:ˆ å©ìÒщԔ gpu.0.gpu‚39.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÕÂþô»1‚(:é å©ìÒ¢·Ú• disk.sda2.out‚0$ memory_percent‚97.62811660766602 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 446.85546875$ disk.nvme0n1.out‚31905.517578125 disk.rbd1.out‚ 44.1611328125 disk.rbd0.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚22242.72265625! disk.dm-0.out‚31905.517578125" disk.nvme0n1.in‚22242.72265625 disk.sda3.out‚ 446.85546875 disk.rbd1.in‚ 2.01953125) disk./.usagePercent‚39.29819384841339& disk.nvme0c0n1.out‚31905.517578125) proc.memory.percent‚17.25475788116455 network.recv‚ 3.650219e+06 disk.sda1.out‚0 disk.dm-1.out‚ 446.85546875 disk.dm-0.in‚22242.72265625# proc.memory.rssMB‚ 2827.01953125 network.sent‚ 2.5443831e+07 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125$ disk./.usageGB‚585.5714874267578 disk.sda3.in‚ 0.1328125& proc.memory.availableMB‚ 388.609375 cpu‚26.207725737650705 proc.cpu.threads‚17”ÛH›¼1‚(: ô©ìÒÀÛÓ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ òŒò½1‚(:ç ô©ìÒçÀï•& disk.nvme0c0n1.out‚31943.599609375# proc.memory.rssMB‚ 2827.01953125 disk.rbd0.in‚0 disk.sda.out‚ 448.7421875 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚22242.72265625) disk./.usagePercent‚39.30619382785087 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.25475788116455 disk.sda3.out‚ 448.7421875 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 44.2236328125 disk.sda1.in‚0 disk.dm-1.out‚ 448.7421875 disk.dm-0.in‚22242.72265625 disk.sda3.in‚ 0.1328125! disk.dm-0.out‚31943.599609375$ memory_percent‚97.63085842132568 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚22242.72265625 disk.sda.in‚ 0.1328125( proc.memory.availableMB‚ 388.16015625 cpu‚26.20718836316407 network.sent‚ 2.5448864e+07 network.recv‚ 3.653424e+06$ disk./.usageGB‚585.6906929016113$ disk.nvme0n1.out‚31943.599609375 proc.cpu.threads‚17Ô àñ4¾1/ _wandbruntime‚13695  _runtime‚13695|˜gÌ¿1Æ ' lr-AdamW/pg1‚4.5694114642693294e-05 ' lr-AdamW/pg2‚4.5694114642693294e-05 trainer/global_step‚15949 % _timestamp‚1.7843540367771413e+09 _runtime‚13695.470693486 _step‚1311TC#JÑÀ1Ë ' lr-AdamW/pg1‚4.5694114642693294e-05 ' lr-AdamW/pg2‚4.5694114642693294e-05 trainer/global_step‚15949 % _timestamp‚1.7843540367771413e+09 _runtime‚13695.470693486 _step‚1311Ÿ ÒöEd4Á1/ _wandbruntime‚13695  _runtime‚13695ºXø'sÂ1€P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2428156667732853 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147719578 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27936908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550977336537600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16864051109376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274159814 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7843540367782128e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2423783362977792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚13695.47189769 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126440236 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147709601 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2423783345938432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34262337 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2423783345938432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27936911 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533933249292288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2440647397047808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17044200726016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17044105857536 ZLDeviceStatsMonitor.Ç)òA“on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16864392448512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2428157025322103 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16838659798135 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2423783362977792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚13695.47189769 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126440236 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147709601 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2423783345938432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34262337 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2423783345938432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27936911 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533933249292288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2440647397047808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17044200726016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17044105857536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16864392448512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2428157025322103 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16838659798135 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126447868 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2423930604007424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17045217547264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411464859156480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62202984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16839330301967 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126447868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126447866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34264331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551011788416000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551011895598080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17045128977408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2428304189458447 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚13696.085722558 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2440796009090048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126447866 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16865405082624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147728644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16839670851969 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2440795650704384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274166519 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147718653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34264607 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27938656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147718653 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27938653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ãã;ã'Æ1N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533966678050816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126447868 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2423930604007424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17045217547264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411464859156480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62202984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16839330301967 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126447868 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126447866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34264331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551011788416000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551011895598080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17045128977408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2428304189458447 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚13696.085722558 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2440796009090048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126447866 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16865405082624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147728644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16839670851969 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2440795650704384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274166519 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147718653 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34264607 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27938656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147718653 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27938653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2423930604007424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16865063736320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _step‚1313 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2428304547047809 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2411464876195840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2440796009090048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16865063736320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚15949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147728644 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274176512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16865405082624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843540373919384e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62203263 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77¡ "GW4Ç1/ _wandbruntime‚13696  _runtime‚13696²º¢È1‰ /loss/FRt1/detection_loss‚0.5340866647939448 /loss/FBt1/detection_loss‚0.3720555028018181 3loss/FBt2/assignment_loss‚0.0014911505762443492 /l?ùv5moss/FRt2/detection_loss‚1.2581867680162153 trainer/global_step‚15949 % _timestamp‚1.7843540373953533e+09 _runtime‚13696.088637171 3loss/SRqqt1/assignment_loss‚0.46125226173305633 4loss/FBt1/assignment_loss‚1.7563642309426086e-05 /loss/FBt2/detection_loss‚0.2954628072257037 %loss/total_loss‚6.921587735255816 _step‚1314 0loss/FRt1/assignment_loss‚1.4403547233462208 0loss/FRt2/assignment_loss‚0.7237740992257993 1loss/SRqqt2/assignment_loss‚0.468886109659688 1loss/SRqqt1/detection_loss‚0.7641395551314496 0loss/SRqqt2/detection_loss‚0.601880529103367 epoch‚39¯ø‡É1Ž _step‚1314 0loss/FRt1/assignment_loss‚1.4403547233462208 0loss/FRt2/assignment_loss‚0.7237740992257993 1loss/SRqqt2/assignment_loss‚0.468886109659688 1loss/SRqqt1/detection_loss‚0.7641395551314496 0loss/SRqqt2/detection_loss‚0.601880529103367 epoch‚39 /loss/FRt1/detection_loss‚0.5340866647939448 /loss/FBt1/detection_loss‚0.3720555028018181 3loss/FBt2/assignment_loss‚0.0014911505762443492 /loss/FRt2/detection_loss‚1.2581867680162153 trainer/global_step‚15949 % _timestamp‚1.7843540373953533e+09 _runtime‚13696.088637171 3loss/SRqqt1/assignment_loss‚0.46125226173305633 4loss/FBt1/assignment_loss‚1.7563642309426086e-05 /loss/FBt2/detection_loss‚0.2954628072257037 %loss/total_loss‚6.921587735255816¢ Œ¯æžÊ1‚(:“ ƒªìÒ†³š” gpu.0.gpu‚93.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û²oøË1‚(:í ƒªìÒû¾Ç˜ disk.dm-0.in‚22242.72265625) disk./.usagePercent‚39.30663160073442 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚26.206642140680703 disk.dm-1.out‚ 449.5078125 disk.rbd1.in‚ 2.01953125 disk.sda.out‚ 449.5078125# proc.memory.rssMB‚ 2827.01953125 disk.drbd2165.out‚0 disk.rbd1.out‚ 44.2236328125$ memory_percent‚97.63379096984863' proc.memory.availableMB‚ 387.6796875) proc.memory.percent‚17.25475788116455" disk.nvme0n1.in‚22242.72265625 network.recv‚ 3.656922e+06$ disk.nvme0c0n1.in‚22242.72265625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚31958.13916015625# disk.dm-0.out‚31958.13916015625 disk.sda3.out‚ 449.5078125 network.sent‚ 2.5466754e+07$ disk./.usageGB‚585.6972160339355 disk.sda1.in‚0& disk.nvme0n1.out‚31958.13916015625 disk.sda3.in‚ 0.1328125 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.rbd0.out‚0@“Ì1‚(:ˆ ’ªìÒÏܵ“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]þëÜøÍ1‚(:í ’ªìÒæâ‘— cpu‚26.206095230883708 proc.cpu.threads‚17 disk.sda.out‚ 449.9609375 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚22242.72265625 disk.sda.in‚ 0.1328125$ disk./.usageGB‚585.6502075195312$ disk.nvme0c0n1.in‚22242.72265625# disk.dm-0.out‚32021.50927734375 disk.rbd1.in‚ 2.01953125 disk.dm-1.out‚ 449.9609375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 449.9609375 disk.rbd1.out‚ 44.2861328125 disk.rbd0.in‚0 network.sent‚ 2.5516925e+07" disk.nvme0n1.in‚22242.72265625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2827.01953125 disk.sda1.in‚0$ memory_percent‚97.63360023498535' proc.memory.availableMB‚ 387.7109375) proc.memory.percent‚17.25475788116455 network.recv‚ 3.661261e+06( disk.nvme0c0n1.out‚32021.50927734375 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0) disk./.usagePercent‚39.30347681988998 disk.drbd2165.out‚0& disk.nvme0n1.out‚32021.50927734375Ü"ao4Î1/ _wandbruntime‚13735  _runtime‚13735ZDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2431145397420032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17095297576960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚13735.49305436 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16915087950336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126821836 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2448060144031232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16889276285043 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552699966183424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34365869 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16888935735153 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2435532817813361 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2418643899117568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2431145414459392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148164053 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126821836 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148174030 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843540767993367e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2448060502409728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16914746611200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62390030 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62389767 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34365609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2448060502409728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552700073372672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15999 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126821834 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2431145414459392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2435533175402611 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28024161 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535604668606464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16915087950336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148164053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995866 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274985887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2418643882078208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2431145397420032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148174030 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126821834 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535604687218688 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2431145397420032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17095297576960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚13735.49305436 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16915087950336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126821836 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2448060144031232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16889276285043 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552699966183424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34365869 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16888935735153 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2435532817813361 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2418643899117568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2431145414459392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148164053 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126821836 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148174030 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843540767993367e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2448060502409728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16914746611200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62390030 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62389767 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34365609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2448060502409728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552700073372672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15999 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126821834 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2431145414459392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2435533175402611 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28024161 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535604668606464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16915087950336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148164053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995866 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274985887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2418643882078208¤ Ä’Ì04Ô1/ _wandbruntime‚13736  _runtime‚13736ŒˆµA²Õ1‹N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34367676 % _timestamp‚1.7843540774128137e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2431292655489024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17096410886656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62393579 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17096316025344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16890287338877 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275012564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275002571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28025903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535638115977216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62393845 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚13736.106564318 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2431292655489024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2431292638449664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16915759238144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126829468 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34367939 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2431292638449664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2435680697128317 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126829466 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148183096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16915759238144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552734413390336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148173105 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16916100584448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126829466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552734526863872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2418790392750080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_byteK³_ s.all.freed‚2448208397687808 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2418790409789440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16889946788875 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275002571 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148183096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2448208756073472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16916100584448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126829468 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28025906 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126829468 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28025906 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62393579 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17096316025344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16890287338877 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275012564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275002571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28025903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535638115977216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62393845 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚13736.106564318 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2431292655489024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2431292638449664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16915759238144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126829468 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34367939 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2431292638449664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2435680697128317 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126829466 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148183096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16915759238144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚15999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552734413390336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148173105 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16916100584448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126829466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552734526863872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2418790392750080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2448208397687808 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2418790409789440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16889946788875¥ 0€þµ4×1/ _wandbruntime‚13736  _runtime‚13736¨%$Ø1‰ _runtime‚13736.109761134 3loss/SRqqt2/assignment_loss‚0.42419188527985224 3loss/FBt1/assignment_loss‚6.816339093647499e-06 /loss/FRt2/detection_loss‚1.2882734939559684 1loss/SRqqt1/detection_loss‚0.9132915122304545 trainer/global_step‚15999 _step‚1318 epoch‚39 /loss/FRt2/assignment_loss‚0.712303373743506 /loss/FRt1/detection_loss‚0.5809668248352894 /loss/FBt2/detection_loss‚0.3362572881695884 1loss/SRqqt2/detection_loss‚0.6055113111792839 % _timestamp‚1.7843540774164433e+09 0loss/FRt1/assignment_loss‚1.5325183176732324 2loss/SRqqt1/assignment_loss‚0.4607132951046978 3loss/FBt2/assignment_loss‚0.0002949702204424195 /loss/FBt1/detection_loss‚0.7680212626371279 %loss/total_loss‚7.622350351368537’PèfÙ1Ž 0loss/FRt1/assignment_loss‚1.5325183176732324 2loss/SRqqt1/assignment_loss‚0.4607132951046978 3loss/FBt2/assignment_loss‚0.0002949702204424195 /loss/FBt1/detection_loss‚0.7680212626371279 %loss/total_loss‚7.622350351368537 _runtime‚13736.109761134 3loss/SRqqt2/assignment_loss‚0.42419188527985224 3loss/FBt1/assignment_loss‚6.816339093647499e-06 /loss/FRt2/detection_loss‚1.2882734939559684 1loss/SRqqt1/detection_loss‚0.9132915122304545 trainer/global_step‚15999 _step‚1318 epoch‚39 /loss/FRt2/assignment_loss‚0.712303373743506 /loss/FRt1/detection_loss‚0.5809668248352894 /loss/FBt2/detection_loss‚0.3362572881695884 1loss/SRqqt2/detection_loss‚0.6055113111792839 % _timestamp‚1.7843540774164433e+09¦ Éî 1“Ú1‚(:ˆ ¡ªìÒè¨ù– gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü˜›øìÛ1‚(:á ¡ªìÒô’ªš disk.sda3.out‚ 450.53125( disk.nvme0c0n1.out‚32025.11669921875 disk.sda2.in‚0 network.sent‚ 2.553483e+07# disk.nvme0c0n1.in‚ 22335.0078125 disk.rbd1.out‚ 44.2861328125 network.recv‚ 3.664436e+06 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 22335.0078125 disk.sda.in‚ 0.1328125 disk.sda2.out‚0# proc.memory.rssMB‚ 2827.01953125$ memory_percent‚97.63240814208984 disk.sda3.in‚ 0.1328125 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.25475788116455 cpu‚26.206638699576672$ disk./.usageGB‚585.6502151489258 disk.sda1.out‚0 disk.drbd2165.out‚0% proc.memory.availableMB‚ 387.90625 proc.cpu.threads‚17) disk./.usagePercent‚39.30347733190504 disk.dm-1.out‚ 450.53125# disk.dm-0.out‚32025.11669921875 disk.sda.out‚ 450.53125! disk.nvme0n1.in‚ 22335.0078125 disk.dm-1.in‚ 0.1328125& disk.nvme0n1.out‚32025.11669921875µŸl4Ü1/ _wandbruntime‚13753  _runtime‚13753™“ʱ=Ý1«{ 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6677655577659607 particle/accuracy_2_of_4‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24599453806877136 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7676330804824829 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6421334147453308 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5078052282333374 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868590831756592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6563608050346375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.272772878408432 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36726126074790955 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7012566328048706 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7345499992370605 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7814322113990784 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 *jet/accuracy_1_of_3‚0.9250309467315674 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6761994957923889 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6463370323181152 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7209402918815613 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15175870060920715 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01908133737742901 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7867608666419983 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943944215774536 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8324769139289856 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6814889311790466 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8787792921066284 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13543321192264557 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01908133737742901 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7992441058158875 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.272772878408432 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7254995703697205 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2360992729663849 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03549830988049507 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2194095104932785 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2838287949562073 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7718461155891418 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6805304288864136 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6393094658851624 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10496141016483307 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27159807085990906 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚13753.232370381 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09698943048715591 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2903302311897278 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5757100582122803 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5035597681999207 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5366523861885071 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 )jet/accuracy_2_of_4‚0.761794924736023 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚1 trainer/global_step‚15999 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *particle/sensitivity‚0.592779278755188 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33018195629119873 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7validation_average_jet_accuracy‚0.37239283323287964 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27350735664367676 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6991100907325745 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚1319 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8530234098434448 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8887377381324768 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3377847373485565 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3062240183353424 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22933292388916016 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2726880609989166 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.241400808095932 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8704361915588379 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚7.685739921312068 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35485517978668213 *particle/specificity‚0.898033857345581 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19388125836849213 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03545500710606575 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_3_of_3‚0.0393088273704052 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24147622287273407 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2843407094478607 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20558354258537292 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023611968383193016 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6409088373184204 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5525997877120972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044410496950149536 +jet/accuracy_3_of_4‚0.04078979045152664 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6548267006874084 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025305751711130142 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7060478925704956 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6031357645988464 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025037596002221107 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7879371643066406 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1299000084400177 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023790793493390083 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6141048669815063 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8704361915588379 *jet/accuracy_1_of_4‚0.9780295491218567 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954846501350403 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5723982453346252 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957804083824158 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744406461715698 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994900107383728 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.621688723564148 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49493956565856934 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 epoch‚39 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15394026041030884 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5975913405418396 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8504912853240967 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860764741897583 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7663648724555969 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7434982657432556 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.050025030970573425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 $ _timestamp‚1.784354094538094e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23056361079216003 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20548860728740692 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24218003451824188 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8787792921066284 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5545833706855774 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6431916952133179 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2647329866886139 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036224640905857086 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7816045880317688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7785289287567139 particle/accuracy_3_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8839566111564636 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7209208011627197 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32104113698005676 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08593904972076416 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16700594127178192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2386288344860077 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1503397673368454 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25607413053512573 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7923046350479126 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7016425132751465 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467148780822754 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7062633037567139 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938677549362183 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_2‚0.1271083652973175 /particle/accuracy_4_of_4‚0.9476639032363892 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5701842308044434 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6365639567375183 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6227486729621887 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6216887235641487ÀIÝŠÞ1°{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7663648724555969 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7434982657432556 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.050025030970573425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 $ _timestamp‚1.784354094538094e+09 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23056361079216003 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20548860728740692 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24218003451824188 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8787792921066284 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5545833706855774 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.99801582 ‘Ö,90980072 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6431916952133179 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2647329866886139 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036224640905857086 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7816045880317688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7785289287567139 particle/accuracy_3_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8839566111564636 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7209208011627197 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32104113698005676 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08593904972076416 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16700594127178192 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2386288344860077 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1503397673368454 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25607413053512573 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7923046350479126 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7016425132751465 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467148780822754 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7062633037567139 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938677549362183 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_2‚0.1271083652973175 /particle/accuracy_4_of_4‚0.9476639032363892 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5701842308044434 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6365639567375183 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6227486729621887 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.621688723564148 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6677655577659607 particle/accuracy_2_of_4‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24599453806877136 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7676330804824829 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6421334147453308 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5078052282333374 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868590831756592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6563608050346375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.272772878408432 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36726126074790955 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7012566328048706 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7345499992370605 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7814322113990784 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 *jet/accuracy_1_of_3‚0.9250309467315674 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6761994957923889 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6463370323181152 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7209402918815613 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15175870060920715 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01908133737742901 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7867608666419983 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943944215774536 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8324769139289856 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6814889311790466 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8787792921066284 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13543321192264557 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01908133737742901 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7992441058158875 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.272772878408432 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7254995703697205 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2360992729663849 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03549830988049507 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2194095104932785 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2838287949562073 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7718461155891418 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6805304288864136 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6393094658851624 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10496141016483307 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27159807085990906 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚13753.232370381 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09698943048715591 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2903302311897278 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5757100582122803 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5035597681999207 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5366523861885071 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 )jet/accuracy_2_of_4‚0.761794924736023 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚1 trainer/global_step‚15999 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *particle/sensitivity‚0.592779278755188 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33018195629119873 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7validation_average_jet_accuracy‚0.37239283323287964 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27350735664367676 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6991100907325745 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚1319 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8530234098434448 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8887377381324768 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3377847373485565 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3062240183353424 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22933292388916016 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2726880609989166 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.241400808095932 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8704361915588379 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚7.685739921312068 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35485517978668213 *particle/specificity‚0.898033857345581 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19388125836849213 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03545500710606575 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_3_of_3‚0.0393088273704052 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24147622287273407 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2843407094478607 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20558354258537292 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023611968383193016 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6409088373184204 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5525997877120972 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044410496950149536 +jet/accuracy_3_of_4‚0.04078979045152664 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6548267006874084 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025305751711130142 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7060478925704956 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6031357645988464 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025037596002221107 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7879371643066406 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1299000084400177 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023790793493390083 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6141048669815063 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8704361915588379 *jet/accuracy_1_of_4‚0.9780295491218567 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954846501350403 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5723982453346252 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957804083824158 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7744406461715698 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994900107383728 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.621688723564148 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49493956565856934 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 epoch‚39 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15394026041030884 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5975913405418396 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8504912853240967 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860764741897583 ("Purity/2FRt1SRqqt1FBt/event_purity‚0§ Élõ3ß1‚(:’ °ªìÒýó¤” gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚207.349' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚41.46979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•RÛ¿øà1‚(:í °ªìÒӾȕ# disk.dm-0.out‚32029.85302734375 disk.sda2.in‚0) proc.memory.percent‚17.25475788116455 network.sent‚ 2.5584999e+07! disk.nvme0n1.in‚ 22335.0078125& disk.nvme0n1.out‚32029.85302734375# memory_percent‚97.6334810256958( proc.memory.availableMB‚ 387.73046875# proc.memory.rssMB‚ 2827.01953125 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚26.209951422109324 proc.cpu.threads‚17( disk.nvme0c0n1.out‚32029.85302734375 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 450.73046875 disk.sda1.out‚0 disk.sda3.in‚ 0.1328125 network.recv‚ 3.668816e+06$ disk./.usageGB‚585.6502456665039 disk.dm-1.in‚ 0.1328125 disk.sda3.out‚ 450.73046875 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 44.4111328125) disk./.usagePercent‚39.30347937996532 disk.dm-1.out‚ 450.73046875# disk.nvme0c0n1.in‚ 22335.0078125 disk.dm-0.in‚ 22335.0078125 disk.sda.in‚ 0.1328125 disk.rbd0.out‚0 disk.sda1.in‚0 ]JÒžá1‚(:“ ¿ªìÒ€´ã“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%)ùâ1‚(:î ¿ªìÒº˜’– disk.dm-0.in‚ 22335.0078125 disk.sda.out‚ 451.30078125 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0# disk.dm-0.out‚32033.88916015625 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚32033.88916015625 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 44.4111328125$ disk./.usageGB‚585.6503257751465 disk.dm-1.out‚ 451.30078125! disk.nvme0n1.in‚ 22335.0078125# proc.memory.rssMB‚ 2827.01953125 cpu‚26.209412904600644 disk.sda2.in‚0 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 451.30078125( proc.memory.availableMB‚ 387.66796875 network.sent‚ 2.5642827e+07 disk.rbd0.in‚0) proc.memory.percent‚17.25475788116455 proc.cpu.threads‚17$ memory_percent‚97.63386249542236 network.recv‚ 3.673268e+06) disk./.usagePercent‚39.30348475612354 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 22335.0078125( disk.nvme0c0n1.out‚32033.88916015625 disk.sda1.out‚0 disk.drbd2165.out‚0l†ÿã1‚(:’ ΪìÒÜùË“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vë¥iôä1‚(:é ΪìÒ긎– disk.sda3.out‚ 451.69140625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 22335.0078125' disk.nvme0c0n1.out‚32039.2314453125$ memory_percent‚97.63267040252686 cpu‚26.20887798944224 proc.cpu.threads‚17$ disk./.usageGB‚585.6504898071289# proc.memory.rssMB‚ 2827.01953125 network.sent‚ 2.564786e+07 disk.sda2.out‚0% disk.nvme0n1.out‚32039.2314453125) proc.memory.percent‚17.25475788116455 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 44.4736328125 disk.dm-1.out‚ 451.69140625 disk.drbd2165.out‚0) disk./.usagePercent‚39.30349576444751 disk.sda2.in‚0! disk.nvme0n1.in‚ 22335.0078125 disk.rbd1.in‚ 2.01953125 disk.dm-1.in‚ 0.1328125( proc.memory.availableMB‚ 387.86328125 network.recv‚ 3.676641e+06 disk.sda.out‚ 451.69140625 disk.drbd2165.in‚0 disk.dm-0.in‚ 22335.0078125 disk.sda.in‚ 0.1328125 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0" disk.dm-0.out‚32039.2314453125ò¤†¼4å1/ _wandbruntime‚13792  _runtime‚13792o´÷Êæ1Ä & lr-AdamW/pg1‚4.354106214373493e-05 & lr-AdamW/pg2‚4.354106214373493e-05 trainer/global_step‚16049 % _timestamp‚1.7843541339491823e+09 _runtime‚13792.642790118 _step‚1320Î=œÈÏç1É & lr-AdamW/pg2‚4.354106214373493e-05 trainer/global_step‚16049 % _timestamp‚1.7843541339491823e+09 _runtime‚13792.642790118 _step‚1320 & lr-AdamW/pg1‚4.354106214373493e-05¨ rã4è1/ _wandbruntime‚13792  _runtime‚13792¬·õD(é1€P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2457420190870528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 $ _timestamp‚1.784354133950314e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444835708422428 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444835350833178 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022906 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62616336 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148698804 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16983175318528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164833253888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440437015552000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537292037244928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148698804 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62616074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537292018632704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28113931 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276012927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34502405 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2457420190870528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16983516657664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554456870498816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16957612497180 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚1321 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554456767503872 _runtime‚13792.644014189 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17164748871168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2427878095925248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127324100 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979ÝÜxµ ê1…P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537292018632704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28113931 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276012927 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34502405 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2457420190870528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16983516657664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554456870498816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16957612497180 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚1321 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554456767503872 _runtime‚13792.644014189 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17164748871168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2427878095925248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127324100 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2457420190870528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 $ _timestamp‚1.784354133950314e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444835708422428 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444835350833178 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022906 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62616336 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148698804 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16983175318528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164833253888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440437015552000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537292037244928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148698804 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62616074© Ž· b4ë1/ _wandbruntime‚13793  _runtime‚13793]­'Ñ'ì1ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127331732 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62619903 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34504485 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2444982872558772 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62620161 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28115673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148707870 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2440584273620992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2428024589557760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1322 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276039604 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784354134564238e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2457568444527104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16958283001012 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276029611 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34504230 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16984187945472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127331734 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2457568802912768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16984187945472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428024606597120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2457568802912768 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276039604 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784354134564238e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2457568444527104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16958283001012 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276029611 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34504230 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16984187945472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127331734 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2457568802912768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16984187945472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428024606597120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2457568802912768 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62619903 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34504485 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2444982872558772 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62620161 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28115673 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148707870 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2440584273620992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2428024589557760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1322 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2447799067033600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556179805647360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2464832937853952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843541737719138e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538963456559104 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2447799067033600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17033870820352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452211858502936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1325 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127705702 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2447799084072960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚16099 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17008228984088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62808404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2435203629518848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127705700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34607223 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149153256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538963475171328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149153256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0%(׸ (ü1†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2435203629518848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127705700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34607223 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149153256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538963475171328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149153256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2464833296232448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127705700 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149143279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34606964 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2464832937853952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2452211500913686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚13832.465654967 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17216330476032 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2464833296232448 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2447799084072960 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2447799067033600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556179805647360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2464832937853952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843541737719138e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538963456559104 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2447799067033600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17033870820352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452211858502936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1325 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127705702 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2447799084072960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚16099 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17008228984088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62808404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269­ cM¶4ý1/ _wandbruntime‚13833  _runtime‚13833øLa 'þ1ŠN E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34608986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2447946308063232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149152331 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚556214164549632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2435350140190720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276865663 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127713334 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149162322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34609247 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2464981549896192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17217279232000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28202923 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17217361510400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17035224793600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28202926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2452359022639280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276865663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62811909 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784354174384993e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62812173 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538996903929856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2452359380228642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚16099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127713332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2435350123151360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17009240037922 _step‚1326 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127713334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2464981549896192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2464981191510528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538996885317632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149152331 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17034883447296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJD¶üh¨"eviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17217279232000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28202923 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17217361510400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17035224793600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28202926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2452359022639280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276865663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62811909 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784354174384993e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62812173 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538996903929856 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2452359380228642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚16099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127713332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2435350123151360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17009240037922 _step‚1326 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127713334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2464981549896192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2464981191510528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538996885317632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149152331 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17034883447296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34711076 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2472245684837376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17084907661312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2455161118515200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28288428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540634913097728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128087302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149597731 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2472246043215872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557902610770432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442529146073088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚13872.457699816 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128087300 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149607708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149597731 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17267796473344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17267716284928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088—Ø»q!Œ2…P [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17058504921106 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2472245684837376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17084566322176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34711340 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557902709571072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2472245684837376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17084907661312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2455161118515200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28288428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540634913097728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128087302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149597731 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2472246043215872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557902610770432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442529146073088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚13872.457699816 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128087300Bfãš QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149607708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149597731 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17267796473344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17267716284928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2459587650994194 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2442529163112448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17084907661312 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34711076 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701715 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17085920295424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚13873.070236954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540668341856256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2455308359544832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17085578949120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128094932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63003349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34713431 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2459735530309150 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711708 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149606783 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚16149 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17268745815040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17085578949120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2455308359544832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557937069059072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784354214376481e+09 _step‚1330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28290173 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149616774 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2472394296879616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128094932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0JƒøC'2ŽN UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149616774 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2472394296879616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128094932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540668323244032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557937167852544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17059515974828 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277711708 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2442675656744960 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17085920295424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 277701715 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63003607 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701715 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17085920295424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚13873.070236954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540668341856256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2455308359544832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17085578949120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128094932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63003349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34713431 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2459735530309150 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711708 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149606783 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚16149 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17268745815040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17085578949120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2455308359544832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚557937069059072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784354214376481e+09 _step‚1330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28290173 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362² ¯Âÿ42/ _wandbruntime‚13873  _runtime‚13873¹B¶‘2Š trainer/global_step‚16149 0loss/FRt2/assignment_loss‚0.7195180834725846 2loss/SRqqt1/assignment_loss‚0.4374896074942939 /loss/FRt2/detection_loss‚1.2502906735118546 %loss/total_loss‚7.153634991073377 epoch‚40 % _timestamp‚1.7843542143803074e+09 0loss/FRt1/assignment_loss‚1.4403841970108893 2loss/SRqqt2/assignment_loss‚0.4216613640386424 3loss/FBt1/assignment_loss‚1.546526037525353e-05 _step‚1331 4loss/FBt2/assignment_loss‚0.00011335461375216446 /loss/FRt1/detection_loss‚0.5844816240802239 /loss/FBt1/detection_loss‚0.4021643584982928 /loss/FBt2/detection_loss‚0.4404902013573006 _runtime‚13873.073583409 1loss/SRqqt1/detection_loss‚0.8874406016984722 1loss/SRqqt2/detection_loss‚0.5695854600366955« •’2 trainer/global_step‚16149 0loss/FRt2/assignment_loss‚0.7195180834725846 2loss/SRqqt1/assignment_loss‚0.4374896074942939 /loss/FRt2/detection_loss‚1.2502906735118546 %loss/total_loss‚7.153634991073377 epoch‚40 % _timestamp‚1.7843542143803074e+09 0loss/FRt1/assignment_loss‚1.4403841970108893 2loss/SRqqt2/assignment_loss‚0.4216613640386424 3loss/FBt1/assignment_loss‚1.546526037525353e-05 _step‚1331 4loss/FBt2/assignment_loss‚0.00011335461375216446 /loss/FRt1/detection_loss‚0.5844816240802239 /loss/FBt1/detection_loss‚0.4021643584982928 /loss/FBt2/detection_loss‚0.4404902013573006 _runtime‚13873.073583409 1loss/SRqqt1/detection_loss‚0.8874406016984722 1loss/SRqqt2/detection_loss‚0.5695854600366955³ ûi«Y“2‚(:’ ¨«ìÒ´›Š” gpu.0.gpu‚68.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_]Žõ”2‚(:ê ¨«ìÒðˆ·–$ memory_percent‚97.63948917388916 network.recv‚ 3.699201e+06 disk.drbd2165.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125) proc.memory.percent‚17.25475788116455$ disk./.usageGB‚585.6498794555664 disk.rbd0.in‚0 disk.sda.in‚ 0.1328125 disk.sda.out‚ 454.51953125# proc.memory.rssMB‚ 2827.01953125 proc.cpu.threads‚17 disk.dm-1.out‚ 454.51953125 disk.drbd2165.in‚0 disk.sda2.out‚0) disk./.usagePercent‚39.30345480324203 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 22520.0078125% disk.nvme0n1.out‚32073.3623046875# disk.nvme0c0n1.in‚ 22520.0078125' disk.nvme0c0n1.out‚32073.3623046875" disk.dm-0.out‚32073.3623046875( proc.memory.availableMB‚ 386.74609375 cpu‚26.20570860262996 disk.sda3.out‚ 454.51953125 disk.rbd1.out‚ 44.7236328125 network.sent‚ 2.5794408e+07 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 22520.00781253XGg‘•2‚(:† ·«ìÒìˆö“ gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uè©Kî–2‚(:ã ·«ìÒ›´…–$ disk./.usageGB‚585.6498413085938& disk.nvme0n1.out‚32076.77294921875 network.sent‚ 2.5857666e+07 disk.sda.out‚ 454.96875 disk.sda2.out‚0$ memory_percent‚97.63956069946289 network.recv‚ 3.703738e+06) disk./.usagePercent‚39.30345224316669 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 22520.0078125 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2827.01953125 cpu‚26.205183080513983# disk.dm-0.out‚32076.77294921875 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 454.96875 disk.rbd1.in‚ 2.01953125 disk.dm-1.out‚ 454.96875( disk.nvme0c0n1.out‚32076.77294921875 disk.sda1.in‚0 disk.dm-0.in‚ 22520.0078125! disk.nvme0n1.in‚ 22520.0078125 disk.rbd1.out‚ 44.7236328125) proc.memory.percent‚17.25475788116455 disk.rbd0.out‚0& proc.memory.availableMB‚ 386.734375 proc.cpu.threads‚17bò“—2‚(:ˆ Æ«ìÒËÓ†” gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?7 ø˜2‚(:í Æ«ìҭȇ—& disk.nvme0n1.out‚32082.38134765625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.1328125# disk.nvme0c0n1.in‚ 22520.0078125# memory_percent‚97.6383924484253 disk.sda.out‚ 455.34765625 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 44.8486328125 network.recv‚ 3.707079e+06$ disk./.usageGB‚585.6499252319336# disk.dm-0.out‚32082.38134765625) disk./.usagePercent‚39.30345787533244 disk.sda2.out‚0 disk.dm-1.out‚ 455.34765625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2827.01953125 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 455.34765625 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.204617369306746! disk.nvme0n1.in‚ 22520.0078125 disk.rbd0.in‚0 disk.dm-0.in‚ 22520.0078125 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚32082.38134765625( proc.memory.availableMB‚ 386.92578125) proc.memory.percent‚17.25475788116455 proc.cpu.threads‚17 network.sent‚ 2.5862551e+07 |Dm4™2/ _wandbruntime‚13912  _runtime‚13912õã§Ìš2Æ ' lr-AdamW/pg1‚4.0398840028278025e-05 ' lr-AdamW/pg2‚4.0398840028278025e-05 trainer/global_step‚16199 % _timestamp‚1.7843542537280715e+09 _runtime‚13912.421672846 _step‚1332]ï&ÍÑ›2Ë _runtime‚13912.421672846 _step‚1332 ' lr-AdamW/pg1‚4.0398840028278025e-05 ' lr-AdamW/pg2‚4.0398840028278025e-05 trainer/global_step‚16199 % _timestamp‚1.7843542537280715e+09´ QMc94œ2/ _wandbruntime‚13912  _runtime‚13912>eþŠ—2€P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17109461957904 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150052183 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559625413753856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2466964158663952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128468902 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479658790199296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128468900 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17318976249856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2449854679666688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150052183 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2462523187036160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17135261824000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559625308661760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2479658431820800 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17135603163136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128468902 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542306332411904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2449854696706048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2462523169996800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17135603163136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278531062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28375678 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17109121408014 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479658431820800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63191021 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ˆj( (ž2…P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128468902 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542306332411904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2449854696706048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2462523169996800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17135603163136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278531062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28375678 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17109121408014 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479658431820800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63191021 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17109461957904 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150052183 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559625413753856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2466964158663952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128468902 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479658790199296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128468900 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17318976249856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2449854679666688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150052183 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2462523187036160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17135261824000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559625308661760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2479658431820800 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17135603163136µ ­ìŒT4Ÿ2/ _wandbruntime‚13913  _runtime‚13913ºŽ*S' 2ŠN UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128476534 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17110132461736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚13913.035766588 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278537767 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2462670428065792 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128476532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150071226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150061235 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150071226 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17320087980544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2467111680389658 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2467111322800296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63194596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559659867763200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542339779782656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34817436 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34817173 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128476534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450001207377920YlAõ¡2N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63194596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559659867763200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542339779782656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34817436 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34817173 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128476534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450001207377920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128476534 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17110132461736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚13913.035766588 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278537767 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2462670428065792 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128476532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150071226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150061235 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150071226 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17320087980544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2467111680389658 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2467111322800296¶ \õÙ4¢2/ _wandbruntime‚13913  _runtime‚13913j½8£2Š 2loss/FBt2/assignment_loss‚0.010097014825983238 1loss/SRqqt1/detection_loss‚0.7429670193316844 0loss/FRt2/assignment_loss‚0.5904240283205753 3loss/SRqqt1/assignment_loss‚0.41675835127940336 /loss/FRt1/detection_loss‚0.5707389127386672 0loss/FBt2/detection_loss‚0.22578505793062137 _step‚1335 0loss/FRt1/assignment_loss‚1.5265348417472637 2loss/SRqqt2/assignment_loss‚0.3769939637280568 /loss/FRt2/detection_loss‚1.1469931443360282 /loss/FBt1/detection_loss‚0.2653814332276982 %loss/total_loss‚6.408910304567598 epoch‚40 trainer/global_step‚16199 3loss/FBt1/assignment_loss‚5.028936334485122e-05 1loss/SRqqt2/detection_loss‚0.5361862477382725 % _timestamp‚1.7843542543459911e+09 _runtime‚13913.039315371 X()¤2 0loss/FRt2/assignment_loss‚0.5904240283205753 3loss/SRqqt1/assignment_loss‚0.41675835127940336 /loss/FRt1/detection_loss‚0.5707389127386672 0loss/FBt2/detection_loss‚0.22578505793062137 _step‚1335 0loss/FRt1/assignment_loss‚1.5265348417472637 2loss/SRqqt2/assignment_loss‚0.3769939637280568 /loss/FRt2/detection_loss‚1.1469931443360282 /loss/FBt1/detection_loss‚0.2653814332276982 %loss/total_loss‚6.408910304567598 epoch‚40 trainer/global_step‚16199 3loss/FBt1/assignment_loss‚5.028936334485122e-05 1loss/SRqqt2/detection_loss‚0.5361862477382725 % _timestamp‚1.7843542543459911e+09 _runtime‚13913.039315371 2loss/FBt2/assignment_loss‚0.010097014825983238 1loss/SRqqt1/detection_loss‚0.7429670193316844· Ãö“‹“¥2‚(:ˆ Õ«ìÒò›Å“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ÂÉø¦2‚(:í Õ«ìÒž¬Ü•' proc.memory.availableMB‚ 386.5234375$ disk./.usageGB‚585.6498794555664 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 22606.8203125 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2827.01953125 disk.sda3.out‚ 455.69921875( disk.nvme0c0n1.out‚32085.70654296875 network.recv‚ 3.710822e+06) disk./.usagePercent‚39.30345480324203 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 22606.8203125 cpu‚26.204097812655792# disk.dm-0.out‚32085.70654296875& disk.nvme0n1.out‚32085.70654296875 disk.sda3.in‚ 0.1328125) proc.memory.percent‚17.25475788116455 disk.rbd1.out‚ 44.8486328125# disk.nvme0c0n1.in‚ 22606.8203125 network.sent‚ 2.5867567e+07 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 455.69921875 disk.sda.out‚ 455.69921875$ memory_percent‚97.64084815979004 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125ûjø‘§2‚(:† ä«ìÒк­“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ø}Íî¨2‚(:ã ä«ìҽϙ— disk.sda.out‚ 456.234375 disk.dm-1.out‚ 456.234375$ disk./.usageGB‚585.6498184204102 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.095495223999023 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚96.70028686523438 disk.sda3.in‚ 0.1328125# disk.dm-0.out‚32091.08740234375# proc.memory.availableMB‚540.625 disk.sda3.out‚ 456.234375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚32091.08740234375 disk.sda2.out‚0" proc.memory.rssMB‚ 2637.0859375 cpu‚26.203568463179877* disk./.usagePercent‚39.303450707121485 disk.rbd1.out‚ 44.9111328125 disk.sda.in‚ 0.1328125 proc.cpu.threads‚17 network.recv‚ 3.715041e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 22606.8203125! disk.nvme0n1.in‚ 22606.8203125 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0& disk.nvme0n1.out‚32091.08740234375 network.sent‚ 2.593062e+07# disk.nvme0c0n1.in‚ 22606.8203125³²ýZ“©2‚(:ˆ ó«ìÒèÐ÷“ gpu.0.gpu‚36.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~»’³ôª2‚(:é ó«ìÒô‚—) disk./.usagePercent‚39.30345377921189 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.01953125$ memory_percent‚96.69873714447021( disk.nvme0c0n1.out‚32093.81494140625 disk.sda2.out‚0" proc.memory.rssMB‚ 2637.0859375 disk.dm-1.out‚ 456.5234375# disk.dm-0.out‚32093.81494140625 disk.sda2.in‚0( proc.memory.availableMB‚ 540.87890625* proc.memory.percent‚16.095495223999023 proc.cpu.threads‚17 network.recv‚ 3.7179e+06 disk.sda.out‚ 456.5234375# disk.nvme0c0n1.in‚ 22606.8203125 network.sent‚ 2.5934731e+07 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0 disk.sda3.out‚ 456.5234375 disk.sda1.in‚0 disk.dm-0.in‚ 22606.8203125 disk.sda.in‚ 0.1328125 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 44.9111328125 cpu‚26.203064611923466$ disk./.usageGB‚585.6498641967773 disk.rbd0.out‚0! disk.nvme0n1.in‚ 22606.8203125& disk.nvme0n1.out‚32093.81494140625bÉ™$4«2/ _wandbruntime‚13952  _runtime‚13952…úWˬ2Å trainer/global_step‚16249 % _timestamp‚1.7843542935513484e+09 _runtime‚13952.24487863 _step‚1336 ' lr-AdamW/pg1‚3.9374997601382097e-05 ' lr-AdamW/pg2‚3.9374997601382097e-05LU)Э2Ê _step‚1336 ' lr-AdamW/pg1‚3.9374997601382097e-05 ' lr-AdamW/pg2‚3.9374997601382097e-05 trainer/global_step‚16249 % _timestamp‚1.7843542935513484e+09 _runtime‚13952.24487863¸ ?ø¾p4®2/ _wandbruntime‚13952  _runtime‚13952^¾u(¯2P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2457180213260288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17186298664960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2487071178804224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17370174780928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2487071178804224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128850500 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128850500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28462931 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚13952.249311777 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2487071537182720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2487071537182720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17185957325824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150506635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088¦ück (°2†P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2487071537182720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2487071537182720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17185957325824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150506635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2457180213260288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17186298664960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2487071178804224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17370174780928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2487071178804224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128850500 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128850500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28462931 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚13952.249311777¹ #ª#™4±2/ _wandbruntime‚13952  _runtime‚13952Q*×DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2470032479547392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13952.860072134 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128858132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28464676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34920924 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279383812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17187311299072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17161089498646 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128858132 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150525678 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383812 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150525678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63385864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17371292179456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128858134 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279373819 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34921188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150515687 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2487219790846464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚544011217709056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279373819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2474487472880804 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17371209901056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚1338 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2470032462508032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63385597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚544011199096832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150515687 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17187311299072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17160748948644 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2470032479547392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28464673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561382408997888 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2470032462508032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088‡-Ïí'³2N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚561382509888512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2470032462508032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2487219432460800 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2487219790846464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2470032479547392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚13952.860072134 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128858132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28464676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34920924 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279383812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17187311299072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17161089498646 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128858132 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150525678 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383812 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150525678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63385864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17371292179456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128858134 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279373819 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34921188 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150515687 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2487219790846464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚544011217709056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279373819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2474487472880804 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17371209901056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚1338 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2470032462508032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63385597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚544011199096832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150515687 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17187311299072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17160748948644 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2470032479547392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28464673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561382408997888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2477247272960000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129232102 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17236652827648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7843543335892093e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129232100 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2477247289999360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28550181 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚545649226876928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150971064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63572803 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17421482028544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2464505763893248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2494483925787648 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63572803 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17421482028544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2464505763893248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2494483925787648 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2477247272960000½ ÎU¸¹4Á2/ _wandbruntime‚13992  _runtime‚13992CíÕ'Â2‹N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2477394531028992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2477394513989632 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚545682655635456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280219864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚16299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2464652257525760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2477394513989632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17237665454592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2494632537829888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63576344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17422426205184 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2481863622961312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129239734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150980130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63576608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28551923 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150980130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28551926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563105166216192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150970139 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150970139 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17238006800896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2481863980550674 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2494632537829888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17422510580736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13992.899885856 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280209871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129239732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35024421 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17211705985554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563105063228416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129239734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2464652274565120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2494632179444224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843543342060962e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2464652257525760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2477394513989632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17237665454592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2494632537829888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63576344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17422426205184 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2481863622961312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129239734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150980130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63576608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28551923 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150980130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28551926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563105166216192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150970139 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150970139 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17238006800896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2481863980550674 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2494632537829888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17422510580736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13992.899885856 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280209871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209871 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129239732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35024421 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17211705985554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563105063228416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129239734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2464652274565120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219864 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2494632179444224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843543342060962e+09 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¾ “dŸí4Ä2/ _wandbruntime‚13992  _runtime‚13992¨¦\çÅ2Œ /loss/FRt1/detection_loss‚0.5244093697008199 /loss/FBt2/detection_loss‚0.3940115179908129 %loss/total_loss‚6.476224970384433 _runtime‚13992.901570757 3loss/SRqqt1/assignment_loss‚0.38134695081764797 4loss/FBt1/assignment_loss‚2.9367743556192527e-05 0loss/FBt1/detection_loss‚0.20175930552483937 epoch‚40 0loss/FRt2/assignment_loss‚0.7002171108722687 3loss/SRqqt2/assignment_loss‚0.43880049966299356 2loss/FBt2/assignment_loss‚0.010382917806598212 /loss/FRt2/detection_loss‚1.1035624707197422 1loss/SRqqt1/detection_loss‚0.7893280416058234 trainer/global_step‚16299 0loss/FRt1/assignment_loss‚1.3934084307266696 1loss/SRqqt2/detection_loss‚0.5389689872126606 % _timestamp‚1.7843543342082562e+09 _step‚1343“Ü°ÙÆ2‘ _runtime‚13992.901570757 3loss/SRqqt1/assignment_loss‚0.38134695081764797 4loss/FBt1/assignment_loss‚2.9367743556192527e-05 0loss/FBt1/detection_loss‚0.20175930552483937 epoch‚40 0loss/FRt2/assignment_loss‚0.7002171108722687 3loss/SRqqt2/assignment_loss‚0.43880049966299356 2loss/FBt2/assignment_loss‚0.010382917806598212 /loss/FRt2/detection_loss‚1.1035624707197422 1loss/SRqqt1/detection_loss‚0.7893280416058234 trainer/global_step‚16299 0loss/FRt1/assignment_loss‚1.3934084307266696 1loss/SRqqt2/detection_loss‚0.5389689872126606 % _timestamp‚1.7843543342082562e+09 _step‚1343 /loss/FRt1/detection_loss‚0.5244093697008199 /loss/FBt2/detection_loss‚0.3940115179908129 %loss/total_loss‚6.476224970384433¿ ±l+žÇ2‚(:“  ¬ìÒ›Öþ“ gpu.0.gpu‚89.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.235400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,+3à²È2‚(:é  ¬ìÒ­–ò• disk.dm-1.in‚ 0.1328125$ disk.nvme0c0n1.in‚22715.49609375 disk.dm-0.in‚22715.49609375 network.recv‚ 3.729208e+06 disk.sda1.in‚0# x²`B disk.dm-0.out‚32107.23583984375 disk.sda2.in‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.in‚ 2.01953125$ disk./.usageGB‚585.6432151794434 network.sent‚ 2.6007888e+07 disk.dm-1.out‚ 457.828125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚32107.23583984375 disk.sda.in‚ 0.1328125 disk.sda.out‚ 457.828125 disk.rbd1.out‚ 45.0361328125$ memory_percent‚96.70214653015137' proc.memory.availableMB‚ 540.3203125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚22715.49609375" proc.memory.rssMB‚ 2637.0859375 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚32107.23583984375 disk.sda3.out‚ 457.828125 cpu‚26.20152598462223) disk./.usagePercent‚39.30300755807971 disk.rbd0.in‚0* proc.memory.percent‚16.095495223999023sQžäžÉ2‚(:“ ¯¬ìÒÜš• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏðòÙãÊ2‚(:Ø ¯¬ìÒÞà—–$ disk.nvme0c0n1.in‚22715.49609375( proc.memory.availableMB‚ 540.24609375* proc.memory.percent‚16.095495223999023 network.recv‚ 3.733681e+06 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 network.sent‚ 2.6071289e+07 disk.sda.out‚458.25 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.sda3.out‚458.25$ memory_percent‚96.70259952545166" proc.memory.rssMB‚ 2637.0859375$ disk./.usageGB‚586.4217567443848 disk.sda1.out‚0 disk.dm-1.out‚458.25$ disk.nvme0n1.out‚32139.029296875 disk.rbd1.in‚ 2.01953125& disk.nvme0c0n1.out‚32139.029296875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚22715.49609375) disk./.usagePercent‚39.35525613574281! disk.dm-0.out‚32139.029296875 disk.sda1.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚22715.49609375 disk.rbd1.out‚ 45.0361328125 cpu‚26.20100677488691ŽyžË2‚(:“ ¾¬ìÒ«¨¿“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òA²÷úÌ2‚(:ï ¾¬ìÒú±Þ• disk.sda2.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17& proc.memory.availableMB‚ 540.453125 disk.sda2.in‚0 disk.rbd1.out‚ 45.0986328125 network.sent‚ 2.6076286e+07 disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚22715.49609375 disk.sda.out‚ 458.70703125 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚96.70133590698242 network.recv‚ 3.736791e+06$ disk./.usageGB‚586.3952407836914 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 458.70703125 disk.dm-0.in‚22715.49609375 disk.sda.in‚ 0.1328125& disk.nvme0n1.out‚32152.68115234375) disk./.usagePercent‚39.35347662737231 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚32152.68115234375 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 458.70703125* proc.memory.percent‚16.095495223999023 disk.drbd2165.in‚0# disk.dm-0.out‚32152.68115234375" disk.nvme0n1.in‚22715.49609375" proc.memory.rssMB‚ 2637.0859375 cpu‚26.200455999221273dŽË•4Í2/ _wandbruntime‚14032  _runtime‚14032(öËÎ2Å _step‚1344 ' lr-AdamW/pg1‚3.7363055854963905e-05 ' lr-AdamW/pg2‚3.7363055854963905e-05 trainer/global_step‚16349 $ _timestamp‚1.784354373661834e+09 _runtime‚14032.355441742r2æÐÏ2Ê $ _timestamp‚1.784354373661834e+09 _runtime‚14032.355441742 _step‚1344 ' lr-AdamW/pg1‚3.7363055854963905e-05 ' lr-AdamW/pg2‚3.7363055854963905e-05 trainer/global_step‚16349À =Z34Ð2/ _wandbruntime‚14032  _runtime‚14032Ä'"(Ñ2€P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28637428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17261311418628 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129613702 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35125589 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2484609341480960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129613702 _runtime‚14032.356690619 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2484609341480960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2501896672771072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17472481296384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚16349 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564793127487488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17260970868738 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 $ _timestamp‚1.784354373662966e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17287348329472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17287689668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17287348329472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35125867 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2501896672771072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129613700 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151415539 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63763017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547320646191104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151425516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039218 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17472574067712 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2501897031149568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129613700 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151425516 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2489092608905476 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2489092251316226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101„vì (Ò2…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28637428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17261311418628 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129613702 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35125589 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2484609341480960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129613702 _runtime‚14032.356690619 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2484609341480960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2501896672771072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17472481296384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚16349 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564793127487488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17260970868738 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 $ _timestamp‚1.784354373662966e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17287348329472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17287689668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17287348329472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35125867 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2501896672771072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129613700 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151415539 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63763017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547320646191104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151425516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039218 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17472574067712 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2501897031149568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129613700 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151425516 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2489092608905476 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2489092251316226Á ë/h¶4Ó2/ _wandbruntime‚14032  _runtime‚14032AŸ\Ô2‹N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17288702302720 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63766815 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28639176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17473592451584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63767068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2471977791119360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151434582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843543742758005e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2471977808158720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055916 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2484756582510592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129621334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045923 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17288360956416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2489239773041820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2502045284813312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564827583025664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_spdlit_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045923 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2502044926427648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2484756565471232 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129621334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045923 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17288360956416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2489239773041820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2502045284813312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564827583025664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045923 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2502044926427648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2484756565471232 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63766815 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28639176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17473592451584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63767068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2471977791119360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151434582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843543742758005e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2471977808158720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055916 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2484756582510592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ¶Oâ4Ö2/ _wandbruntime‚14032  _runtime‚14032Ô­Æã×2Œ 1loss/SRqqt1/detection_loss‚0.8044878413704425 _step‚1347 0loss/FBt2/detection_loss‚0.35830934518099217 %loss/total_loss‚6.753879737334635 0loss/FRt1/assignment_loss‚1.5190580577231951 4loss/FBt1/assignment_loss‚1.4272088574251996e-05 3loss/FBt2/assignment_loss‚0.0003354156084124454 0loss/FBt1/detection_loss‚0.33010815536399607 trainer/global_step‚16349 % _timestamp‚1.7843543742794921e+09 3loss/SRqqt2/assignment_loss‚0.39422050472884157 .loss/FRt2/detection_loss‚1.178708921083777 epoch‚40 0loss/FRt2/assignment_loss‚0.7019412906996283 3loss/SRqqt1/assignment_loss‚0.38175600183503544 /loss/FRt1/detection_loss‚0.5242070370344647 0loss/SRqqt2/detection_loss‚0.560732894617275 _runtime‚14032.972798392 ·3“Ø2‘ 0loss/FBt2/detection_loss‚0.35830934518099217 %loss/total_loss‚6.753879737334635 0loss/FRt1/assignment_loss‚1.5190580577231951 4loss/FBt1/assignment_loss‚1.4272088574251996e-05 3loss/FBt2/assignment_loss‚0.0003354156084124454 0loss/FBt1/detection_loss‚0.33010815536399607 trainer/global_step‚16349 % _timestamp‚1.7843543742794921e+09 3loss/SRqqt2/assignment_loss‚0.39422050472884157 .loss/FRt2/detection_loss‚1.178708921083777 epoch‚40 0loss/FRt2/assignment_loss‚0.7019412906996283 3loss/SRqqt1/assignment_loss‚0.38175600183503544 /loss/FRt1/detection_loss‚0.5242070370344647 0loss/SRqqt2/detection_loss‚0.560732894617275 _runtime‚14032.972798392 1loss/SRqqt1/detection_loss‚0.8044878413704425 _step‚1347à |™_[“Ù2‚(:ˆ ͬìÒðÝŒ• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y>ì©õÚ2‚(:ê ͬìÒÇ‘ð– disk.dm-0.in‚ 22788.390625 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0 network.recv‚ 3.740511e+06 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.dm-1.in‚ 0.1328125( disk.nvme0c0n1.out‚32941.92041015625 disk.rbd1.in‚ 2.01953125 cpu‚26.199935835246002 proc.cpu.threads‚17$ disk./.usageGB‚585.6512260437012 disk.sda.out‚ 459.08984375 network.sent‚ 2.6081298e+07) disk./.usagePercent‚39.30354517390162# disk.dm-0.out‚32941.92041015625 disk.sda3.out‚ 459.08984375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 539.9453125" proc.memory.rssMB‚ 2637.0859375 disk.sda2.out‚0& disk.nvme0n1.out‚32941.92041015625 disk.rbd0.in‚0 disk.dm-1.out‚ 459.08984375* proc.memory.percent‚16.095495223999023 disk.rbd1.out‚ 45.0986328125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 22788.390625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 22788.390625$ memory_percent‚96.70443534851074V“±’Û2‚(:‡ ܬìÒË¢‹– gpu.0.gpu‚16.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚207.692' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ˆïTöÜ2‚(:ë ܬìÒØ̃˜ disk.rbd1.in‚ 2.01953125 disk.sda1.in‚0 disk.dm-1.out‚ 459.51171875 disk.sda.in‚ 0.1328125 disk.sda3.out‚ 459.51171875 disk.dm-1.in‚ 0.1328125" disk.nvme0c0n1.in‚ 22788.390625 disk.nvme0n1.in‚ 22788.390625 disk.rbd1.out‚ 45.1611328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚32946.89306640625( proc.memory.availableMB‚ 540.24609375) disk./.usagePercent‚39.30355874230093 disk.drbd2165.out‚0* proc.memory.percent‚16.095495223999023$ disk./.usageGB‚585.6514282226562 proc.cpu.threads‚17& disk.nvme0n1.out‚32946.89306640625 disk.dm-0.in‚ 22788.390625 cpu‚26.199434318838858$ memory_percent‚96.70259952545166 disk.sda3.in‚ 0.1328125 disk.rbd0.out‚0# disk.dm-0.out‚32946.89306640625 disk.sda.out‚ 459.51171875 disk.sda2.out‚0" proc.memory.rssMB‚ 2637.0859375 network.sent‚ 2.6146428e+07 network.recv‚ 3.745072e+06 disk.sda2.in‚0س¾¬‘Ý2‚(:† ë¬ìÒáò†• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05T}&èÞ2‚(:Ý ë¬ìÒ¬€ê• disk.dm-0.out‚32950.61328125* proc.memory.percent‚16.095495223999023$ disk./.usageGB‚585.6244773864746 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 539.98046875 cpu‚26.19892924241213 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 45.1611328125" proc.memory.rssMB‚ 2637.0859375 proc.cpu.threads‚17 network.recv‚ 3.749015e+06) disk./.usagePercent‚39.30175004907153 disk.sda1.out‚0 disk.dm-1.in‚ 0.1328125 disk.dm-0.in‚ 22788.390625 disk.sda2.out‚0$ memory_percent‚96.70422077178955 network.sent‚ 2.614959e+07" disk.nvme0c0n1.in‚ 22788.390625 disk.sda.out‚ 459.9296875 disk.nvme0n1.in‚ 22788.390625% disk.nvme0c0n1.out‚32950.61328125 disk.sda2.in‚0 disk.sda3.out‚ 459.9296875 disk.rbd0.out‚0 disk.dm-1.out‚ 459.9296875 disk.sda.in‚ 0.1328125# disk.nvme0n1.out‚32950.61328125 disk.rbd1.in‚ 2.01953125ü8þ÷4ß2/ _wandbruntime‚14072  _runtime‚14072%½Ìà2Æ ' lr-AdamW/pg1‚3.6375085758700505e-05 ' lr-AdamW/pg2‚3.6375085758700505e-05 trainer/global_step‚16399 % _timestamp‚1.7843544134671752e+09 _runtime‚14072.160734573 _step‚1348PØô Ñá2Ë ' lr-AdamW/pg2‚3.6375085758700505e-05 trainer/global_step‚16399 % _timestamp‚1.7843544134671752e+09 _runtime‚14072.160734573 _step‚1348 ' lr-AdamW/pg1‚3.6375085758700505e-05Ä qˆ‡z4â2/ _wandbruntime‚14072  _runtime‚14072Ïéf(ã2P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2509309778132992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2491971375923200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151879968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129995300 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2509309419754496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17338385170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63953100 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548992084117504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28724681 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35228688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2479156831080448 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2491971392962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17338043831296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2491971375923200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129995300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843544134682908e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2509309419754496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548992102729728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2496468401396734 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63953369 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129995302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚14072.162006123 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2479156814041088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2491971392962560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2496468758985984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2491971375923200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151879968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129995300 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2509309419754496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17338385170432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63953100 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548992084117504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28724681 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35228688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2479156831080448 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2491971392962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17338043831296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2491971375923200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129995300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843544134682908e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2509309419754496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548992102729728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2496468401396734 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63953369 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129995302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚14072.162006123 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2479156814041088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269Å KX¬a4å2/ _wandbruntime‚14072  _runtime‚14072â‰J'æ2‹N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2492118633992192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17339056458240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28726426 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚549025531488256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17339397804544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17339056458240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17312938959370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891968 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63956847 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879043 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2492118633992192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28726423 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2492118616952832 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17524568159744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130002934 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151889034 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17524663021056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17312598409368 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2509458031796736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281881975 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2492118616952832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35230675 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151889034 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2509458031796736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130002934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63957101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2496615923122328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17339397804544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2496616280711690 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2479303341752320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881975 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17524568159744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130002934 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151889034 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17524663021056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17312598409368 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2509458031796736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281881975 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2492118616952832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35230675 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151889034 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2509458031796736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130002934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63957101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2496615923122328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17339397804544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2496616280711690 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2479303341752320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881975 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17339056458240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17312938959370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891968 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63956847 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879043 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2492118633992192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28726423 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Æ ËêÉÒ4è2/ _wandbruntime‚14072  _runtime‚14072ê礇é2‰ 1loss/SRqqt2/detection_loss‚0.5828286918564218 % _timestamp‚1.7843544140852625e+09 _runtime‚14072.778571193 3loss/FBt1/assignment_loss‚0.0001431608938033565 0loss/FRt1/assignment_loss‚1.4634711097787927 1loss/SRqqt1/detection_loss‚0.8937652910780775 /loss/FBt1/detection_loss‚0.1300942471023197 /loss/FBt2/detection_loss‚0.3640064844495888 %loss/total_loss‚6.815449439525226 trainer/global_step‚16399 2loss/SRqqt2/assignment_loss‚0.4803210260652966 epoch‚40 _step‚1351 0loss/FRt2/assignment_loss‚0.7056308401515707 3loss/SRqqt1/assignment_loss‚0.49071423266568254 3loss/FBt2/assignment_loss‚0.0009563409586603667 /loss/FRt1/detection_loss‚0.5540774292177938 .loss/FRt2/detection_loss‚1.149440585307218ÿ´Ùê2Ž 0loss/FRt2/assignment_loss‚0.7056308401515707 3loss/SRqqt1/assignment_loss‚0.49071423266568254 3loss/FBt2/assignment_loss‚0.0009563409586603667 /loss/FRt1/detection_loss‚0.5540774292177938 .loss/FRt2/detection_loss‚1.149440585307218 1loss/SRqqt2/detection_loss‚0.5828286918564218 % _timestamp‚1.7843544140852625e+09 _runtime‚14072.778571193 3loss/FBt1/assignment_loss‚0.0001431608938033565 0loss/FRt1/assignment_loss‚1.4634711097787927 1loss/SRqqt1/detection_loss‚0.8937652910780775 /loss/FBt1/detection_loss‚0.1300942471023197 /loss/FBt2/detection_loss‚0.3640064844495888 %loss/total_loss‚6.815449439525226 trainer/global_step‚16399 2loss/SRqqt2/assignment_loss‚0.4803210260652966 epoch‚40 _step‚1351Ç BÅH…žë2‚(:“ ú¬ìÒå‘õ• gpu.0.gpu‚70.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÂõâì2‚(:× ú¬ìÒøÜè›# disk./.usageGB‚585.624439239502 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.out‚32955.76171875 disk.sda2.in‚0 disk.rbd0.in‚0 network.recv‚ 3.752394e+06 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 460.59375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 22788.390625 disk.nvme0n1.in‚ 22788.390625 disk.sda3.in‚ 0.1328125 disk.sda3.out‚ 460.59375 disk.rbd0.out‚0' proc.memory.availableMB‚ 440.8046875 network.sent‚ 2.6167581e+07 disk.sda.in‚ 0.1328125# proc.memory.rssMB‚ 2736.11328125 disk.sda1.in‚0% disk.nvme0c0n1.out‚32955.76171875# disk.nvme0n1.out‚32955.76171875 disk.rbd1.out‚ 45.2236328125* proc.memory.percent‚16.699910163879395) disk./.usagePercent‚39.30174748899618 disk.dm-0.in‚ 22788.390625 disk.sda2.out‚0 disk.rbd1.in‚ 2.01953125$ memory_percent‚97.30954170227051 cpu‚26.20179436633475 disk.sda.out‚ 460.59375Ò4í2/ _wandbruntime‚14089  _runtime‚14089L6c³ î2•{ 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.770160973072052 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5585674047470093 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7850187420845032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951462745666504 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6512817144393921 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5423333048820496 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019212158396840096 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31422170996665955 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.57164466381073 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6474783420562744 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7064847946166992 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8770262598991394 *jet/accuracy_2_of_2‚0.1279841810464859 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7180558443069458 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23606543242931366 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2824288606643677 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047498829662799835 particle/accuracy_1_of_4‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10306505858898163 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_4‚0.9968254566192627 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7637337446212769 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2479291707277298 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29311713576316833 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6821238398551941 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960423707962036 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930713176727295 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7169359922409058 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27422523498535156 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6501791477203369 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6335575580596924 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6324568390846252 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6760219931602478 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.00044ãG$<è017782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9489062428474426 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10133198648691177 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019212158396840096 epoch‚40 _step‚1352 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8986722826957703 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7268273234367371 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.664415717124939 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28249219059944153 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09219275414943695 *jet/accuracy_1_of_2‚0.7845171689987183 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2982493042945862 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21246033906936646 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6057566404342651 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 )loss/val_total_loss‚7.689608821767794 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2935082018375397 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3612605929374695 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7768025398254395 % _timestamp‚1.7843544312666917e+09 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5691941380500793 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25083082914352417 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23871499300003052 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5097325444221497 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6658146977424622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15267030894756317 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02695659175515175 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7923231720924377 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6894856095314026 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7267223000526428 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03747930750250816 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.624317467212677 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18212656676769257 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_2_of_4‚0.7608501315116882 +jet/accuracy_3_of_4‚0.04078979045152664 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7369263768196106 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7458768486976624 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7871834635734558 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 )jet/accuracy_3_of_3‚0.038394995033741 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.150165393948555 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7926210761070251 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9473532438278198 +particle/sensitivity‚0.5954625010490417 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.504677414894104 trainer/global_step‚16399 _runtime‚14089.960867101 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8857157826423645 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024669794365763664 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6544906497001648 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 +jet/accuracy_4_of_4‚0.01670061983168125 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6072461009025574 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2491171658039093 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_1‚0.3447309136390686 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24849119782447815 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9267236590385437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5800313353538513 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23771528899669647 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3742000162601471 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024829106405377388 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3293420374393463 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12554486095905304 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21299302577972412 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6validation_average_jet_accuracy‚0.3782138526439667 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7280880212783813 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2438378781080246 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8770262598991394 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942342042922974 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8270785808563232 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6324568390846252 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7739604711532593 *jet/accuracy_1_of_4‚0.9806751012802124 particle/accuracy_2_of_4‚1 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1670001745223999 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2824288606643677 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22273372113704681 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6884856224060059 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.649160623550415 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7084721922874451 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7909581065177917 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23331083357334137 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007168650627136 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13606776297092438 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026708682999014854 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3373728394508362 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03766250237822533 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05299849808216095 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2805100083351135 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28227466344833374 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6385470032691956 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14420358836650848 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 'particle/accuracy‚0.833558201789856 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8335222601890564 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7112194299697876 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04286383464932442 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8676493763923645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.00083705357974395160çmÄ =ï2š{ 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5800313353538513 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23771528899669647 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3742000162601471 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024829106405377388 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3293420374393463 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12554486095905304 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21299302577972412 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6validation_average_jet_accuracy‚0.3782138526439667 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7280880212783813 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2438378781080246 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8770262598991394 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942342042922974 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998227596282959 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8270785808563232 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6324568390846252 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7739604711532593 *jet/accuracy_1_of_4‚0.9806751012802124 particle/accuracy_2_of_4‚1 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1670001745223999 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2824288606643677 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22273372113704681 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6884856224060059 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.649160623550415 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7084721922874451 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7909581065177917 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23331083357334137 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007168650627136 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13606776297092438 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026708682999014854 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3373728394508362 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03766250237822533 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05299849808216095 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2805100083351135 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28227466344833374 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6385470032691956 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14420358836650848 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 'particle/accuracy‚0.833558201789856 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8335222601890564 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7112194299697876 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04286383464932442 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8676493763923645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977238178253174 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.770160973072052 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5585674047470093 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7850187420845032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951462745666504 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6512817144393921 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5423333048820496 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019212158396840096 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31422170996665955 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.57164466381073 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6474783420562744 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7064847946166992 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8770262598991394 *jet/accuracy_2_of_2‚0.1279841810464859 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7180558443069458 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23606543242931366 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2824288606643677 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047498829662799835 particle/accuracy_1_of_4‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10306505858898163 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_3_of_4‚0.9968254566192627 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7637337446212769 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2479291707277298 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29311713576316833 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998453676700592 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6821238398551941 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960423707962036 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930713176727295 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7169359922409058 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27422523498535156 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6501791477203369 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6335575580596924 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6324568390846252 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6760219931602478 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9489062428474426 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10133198648691177 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019212158396840096 epoch‚40 _step‚1352 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8986722826957703 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7268273234367371 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.664415717124939 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28249219059944153 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09219275414943695 *jet/accuracy_1_of_2‚0.7845171689987183 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2982493042945862 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21246033906936646 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6057566404342651 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 )loss/val_total_loss‚7.689608821767794 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2935082018375397 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3612605929374695 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7768025398254395 % _timestamp‚1.7843544312666917e+09 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5691941380500793 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25083082914352417 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23871499300003052 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5097325444221497 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6658146977424622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15267030894756317 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02695659175515175 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7923231720924377 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6894856095314026 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7267223000526428 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03747930750250816 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.624317467212677 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18212656676769257 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_2_of_4‚0.7608501315116882 +jet/accuracy_3_of_4‚0.04078979045152664 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7369263768196106 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7458768486976624 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7871834635734558 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 )jet/accuracy_3_of_3‚0.038394995033741 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.150165393948555 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7926210761070251 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9473532438278198 +particle/sensitivity‚0.5954625010490417 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.504677414894104 trainer/global_step‚16399 _runtime‚14089.960867101 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8857157826423645 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024669794365763664 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6544906497001648 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 +jet/accuracy_4_of_4‚0.01670061983168125 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6072461009025574 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2491171658039093 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_1‚0.3447309136390686 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24849119782447815 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9267236590385437 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1È ùäážð2‚(:“ ‰­ìÒé…°” gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼­21ïñ2‚(:ä ‰­ìÒì®– disk.sda2.out‚0& disk.nvme0n1.out‚32959.08154296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 460.9609375! disk.nvme0c0n1.in‚ 22798.65625 disk.sda.out‚ 460.9609375$ disk./.usageGB‚585.6244621276855 disk.sda3.out‚ 460.9609375 disk.dm-1.in‚ 0.1328125 disk.drbd2165.out‚0# memory_percent‚97.3494291305542# proc.memory.rssMB‚ 2742.70703125 disk.nvme0n1.in‚ 22798.65625 disk.rbd1.in‚ 2.01953125 disk.rbd1.out‚ 45.2236328125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚32959.08154296875 disk.dm-0.in‚ 22798.65625( proc.memory.availableMB‚ 434.26953125 disk.sda3.in‚ 0.1328125 disk.sda.in‚ 0.1328125 cpu‚26.202751544958446 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚32959.08154296875) proc.memory.percent‚16.74015522003174 network.sent‚ 2.6217751e+07 network.recv‚ 3.757268e+06) disk./.usagePercent‚39.30174902504139‰’ì “ò2‚(:ˆ ˜­ìÒѬ—” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸íßó2‚(:Ô ˜­ìÒËü• disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.01953125& disk.nvme0n1.out‚32966.03466796875 disk.sda3.out‚461.3125( disk.nvme0c0n1.out‚32966.03466796875 disk.drbd2165.out‚0& proc.memory.availableMB‚ 434.359375 network.recv‚ 3.761537e+06 disk.dm-1.in‚ 0.1328125 disk.sda.out‚461.3125 disk.sda2.out‚0 disk.sda3.in‚ 0.1328125 disk.rbd1.out‚ 45.2861328125 proc.cpu.threads‚17) disk./.usagePercent‚39.30174800101125 disk.sda1.out‚0 disk.dm-1.out‚461.3125 disk.sda2.in‚0) proc.memory.percent‚16.74015522003174$ disk./.usageGB‚585.6244468688965 disk.rbd0.out‚0 disk.dm-0.in‚ 22862.125 disk.nvme0n1.in‚ 22862.125 disk.nvme0c0n1.in‚ 22862.125 disk.sda1.in‚0# disk.dm-0.out‚32966.03466796875$ memory_percent‚97.34888076782227# proc.memory.rssMB‚ 2742.70703125 disk.sda.in‚ 0.1328125 cpu‚26.202237435613146 network.sent‚ 2.6275232e+07X{Û4ô2/ _wandbruntime‚14129  _runtime‚14129R *2Ëõ2Å $ _timestamp‚1.784354470698663e+09 _runtime‚14129.392341441 _step‚1353 ' lr-AdamW/pg1‚3.5399201045080246e-05 ' lr-AdamW/pg2‚3.5399201045080246e-05 trainer/global_step‚164499 àÐö2Ê ' lr-AdamW/pg1‚3.5399201045080246e-05 ' lr-AdamW/pg2‚3.5399201045080246e-05 trainer/global_step‚16449 $ _timestamp‚1.784354470698663e+09 _runtime‚14129.392341441 _step‚1353É ¬i%^4÷2/ _wandbruntime‚14129  _runtime‚14129pd.))ø2€P UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2505770934416551 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚14129.397379342 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2501263011094528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282892331 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28814451 XJDeviceStatsMoÂSßãÝ#nitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17379923567783 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17406472538624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17406813877760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568272412088832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2518669466593792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64178655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2518669824972288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568272502500864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130497566 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152394765 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2488391027888128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2488391010848768 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2501262994055168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚16449 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17380264117673 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17406813877760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902310 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130497566 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35363944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130497568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _step‚1354 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152394765 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843544706998098e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2518669466593792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2501262994055168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550679452755968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550679434143744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2501263011094528 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64178392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28814448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17593049744896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2518669824972288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35364204 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152404742 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17406472538624 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2505771292005801 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568272412088832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2518669466593792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64178655 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2518669824972288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568272502500864 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130497566 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152394765 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2488391027888128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2488391010848768 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2501262994055168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚16449 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17380264117673 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17406813877760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902310 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130497566 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35363944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130497568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _step‚1354Ê ZËHP“ú2‚(:ˆ §­ìÒÆæ‚” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚246.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rO¹ê4û2/ _wandbruntime‚14130  _runtime‚14130d]]þ'ü2‹N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17594079435776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2505918813731507 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16449 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130505200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64182253 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130505198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2488537538560000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2501410235084800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28816196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909015 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152403817 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130505198 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152413808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2501410252124160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17407826511872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17380934621505 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2518818078636032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28816193 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2501410235084800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17593995060224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17407485165568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2505918456142145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64182528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚568306857962496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚14130.006753525 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282919008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17381275171507 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17407826511872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568306960950272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282919008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2501410252124160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550712881514496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152403817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2518817720250368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550712862902272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35366332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2488537521520640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282919008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17381275171507 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.largA#Ve_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17407826511872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568306960950272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282919008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2501410252124160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550712881514496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152403817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2518817720250368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550712862902272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35366332 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2488537521520640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16449 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130505200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64182253 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130505198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2488537538560000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2501410235084800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28816196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909015 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152403817 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130505198 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152413808Ë ÂgNhåþ2‚(:Ú §­ìÒШ»– network.recv‚ 3.765278e+06 disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚32970.5185546875 disk.dm-0.in‚ 22862.125& proc.memory.availableMB‚ 434.203125 disk.sda3.out‚ 461.6796875 disk.rbd1.in‚ 2.01953125% disk.nvme0n1.out‚32970.5185546875) proc.memory.percent‚16.74015522003174 disk.sda.out‚ 461.6796875 cpu‚26.20168580799566 proc.cpu.threads‚17 network.sent‚ 2.6280266e+07* disk./.usagePercent‚39.301760545380425 disk.rbd0.in‚0 disk.dm-1.out‚ 461.6796875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 22862.125" disk.dm-0.out‚32970.5185546875 disk.nvme0n1.in‚ 22862.125 disk.dm-1.in‚ 0.1328125# proc.memory.rssMB‚ 2742.70703125 disk.rbd1.out‚ 45.2861328125 disk.sda2.out‚0$ memory_percent‚97.34983444213867$ disk./.usageGB‚585.6246337890625 disk.drbd2165.out‚0¼ t4ÿ2/ _wandbruntime‚14130  _runtime‚14130 ä_]€3‰ 0loss/FRt2/assignment_loss‚0.7595157082282256 /loss/FRt2/detection_loss‚1.2383850861515957 1loss/SRqqt1/detection_loss‚0.7235386960116811 0loss/FBt1/detection_loss‚0.27608662141481033 0loss/FRt1/assignment_loss‚1.3707137559303793 3loss/SRqqt1/assignment_loss‚0.44944448987540736 3loss/SRqqt2/assignment_loss‚0.42106654888098866 2loss/FBt2/assignment_loss‚0.002124874669892798 .loss/FBt2/detection_loss‚0.480640353733386 epoch‚41 2loss/FBt1/assignment_loss‚0.015070958842234024 /loss/FRt1/detection_loss‚0.5839529650887602 1loss/SRqqt2/detection_loss‚0.5954671563527248 %loss/total_loss‚6.916007215180086 trainer/global_step‚16449 % _timestamp‚1.7843544713163855e+09 _runtime‚14130.009675662 _step‚1356Z÷8R3Ž epoch‚41 2loss/FBt1/assignment_loss‚0.015070958842234024 /loss/FRt1/detection_loss‚0.5839529650887602 1loss/SRqqt2/detection_loss‚0.5954671563527248 %loss/total_loss‚6.916007215180086 trainer/global_step‚16449 % _timestamp‚1.7843544713163855e+09 _runtime‚14130.009675662 _step‚1356 0loss/FRt2/assignment_loss‚0.7595157082282256 /loss/FRt2/detection_loss‚1.2383850861515957 1loss/SRqqt1/detection_loss‚0.7235386960116811 0loss/FBt1/detection_loss‚0.27608662141481033 0loss/FRt1/assignment_loss‚1.3707137559303793 3loss/SRqqt1/assignment_loss‚0.44944448987540736 3loss/SRqqt2/assignment_loss‚0.42106654888098866 2loss/FBt2/assignment_loss‚0.002124874669892798 .loss/FBt2/detection_loss‚0.480640353733386Ì CV6I“‚3‚(:ˆ ¶­ìÒÒßµ“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰m+Ùìƒ3‚(:á ¶­ìÒÓÝߘ disk.sda3.in‚ 0.1328125 disk.sda1.out‚0 disk.sda.out‚ 462.33203125& disk.nvme0n1.out‚32977.78369140625 cpu‚26.20118130291766 proc.cpu.threads‚17' proc.memory.availableMB‚ 434.2421875 disk.drbd2165.in‚0 disk.sda.in‚ 0.1328125 disk.sda2.in‚0$ memory_percent‚97.34959602355957 network.sent‚ 2.6332449e+07* disk./.usagePercent‚39.301755169222204 disk.sda3.out‚ 462.33203125 disk.rbd1.out‚ 45.3486328125 disk.sda1.in‚0 disk.nvme0n1.in‚ 22862.125 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 22862.125 disk.dm-0.in‚ 22862.125 disk.sda2.out‚0$ disk./.usageGB‚585.6245536804199( disk.nvme0c0n1.out‚32977.78369140625# disk.dm-0.out‚32977.78369140625 disk.rbd1.in‚ 2.01953125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.1328125 disk.dm-1.out‚ 462.33203125# proc.memory.rssMB‚ 2742.70703125) proc.memory.percent‚16.74015522003174 network.recv‚ 3.769055e+06'LX@“„3‚(:ˆ Å­ìÒÀÉæ“ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚256.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðTí…3‚(:â Å­ìÒ¯ùâ• disk.dm-1.out‚ 463.18359375( disk.nvme0c0n1.out‚32980.44287109375 disk.dm-0.in‚ 22862.125 disk.nvme0n1.in‚ 22862.125 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda.out‚ 463.18359375# proc.memory.rssMB‚ 2742.70703125) proc.memory.percent‚16.74015522003174 proc.cpu.threads‚17) disk./.usagePercent‚39.30175312116194 disk.sda.in‚ 0.16015625 disk.sda2.in‚0& disk.nvme0n1.out‚32980.44287109375 disk.dm-1.in‚ 0.16015625 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 463.18359375& proc.memory.availableMB‚ 434.296875 network.recv‚ 3.773528e+06 disk.rbd0.in‚0 disk.sda3.in‚ 0.16015625 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.34926223754883 network.sent‚ 2.6385047e+07$ disk./.usageGB‚585.6245231628418 disk.sda1.out‚0 disk.rbd1.out‚ 45.3486328125 disk.nvme0c0n1.in‚ 22862.125# disk.dm-0.out‚32980.44287109375 cpu‚26.20068057743214+Ð+A4†3/ _wandbruntime‚14169  _runtime‚14169º~Bkɇ3à _step‚1357 & lr-AdamW/pg1‚3.443546439335477e-05 & lr-AdamW/pg2‚3.443546439335477e-05 trainer/global_step‚16499 % _timestamp‚1.7843545104729996e+09 _runtime‚14169.16655082ið‡BΈ3È & lr-AdamW/pg2‚3.443546439335477e-05 trainer/global_step‚16499 % _timestamp‚1.7843545104729996e+09 _runtime‚14169.16655082 _step‚1357 & lr-AdamW/pg1‚3.443546439335477e-05Í ÌƒB½4‰3/ _wandbruntime‚14169  _runtime‚14169$k‡(Š3€P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569995172526592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35468473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130879166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚552350872070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17457168040448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2526082571955712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2508625062576128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130879168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152849217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2495716544442368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚552350890682368 _step‚1358 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152859194 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2508625062576128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17457509379584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17644391130624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2526082213577216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2495716561481728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28901701 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17457168040448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152859194 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283728383 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17457509379584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64370432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚16499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2526082213577216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2508625045536768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17430540054691 % _timestamp‚1.7843545104740767e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚569995281812992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17430880604581 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64370171 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚14169.16784363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28901698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2508625045536768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176ŠÂÿÓW$‹3…P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17430540054691 % _timestamp‚1.7843545104740767e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚569995281812992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17430880604581 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64370171 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚14169.16784363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28901698 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2508625045536768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569995172526592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35468473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130879166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚552350872070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17457168040448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2526082571955712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2508625062576128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130879168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152849217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2495716544442368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚552350890682368 _step‚1358 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152859194 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2508625062576128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17457509379584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17644391130624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2526082213577216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2495716561481728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28901701 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17457168040448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738362 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152859194 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283728383 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17457509379584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64370432 HBDeviceStatsMonitor.on_train_batch_start/overãI2‹´size_allocations.freed‚0 trainer/global_step‚16499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2526082213577216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2508625045536768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Î ‘²eé4Œ3/ _wandbruntime‚14169  _runtime‚14169ðóWŠ'3‹N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2508772303605760 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570029629819904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2495863055114240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28903443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35470500 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2526230467233792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17645415463936 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2513294606222653 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283755060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130886798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17431551108413 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130886800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚552384300828672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283745067 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17458180667392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17431891658415 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745067 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130886800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17645328991232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1359 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2508772286566400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64373943 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570029734904832 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438܃;j'Ž3N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28903443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35470500 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2526230467233792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17645415463936 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2513294606222653 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283755060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130886798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17431551108413 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130886800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚552384300828672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283745067 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17458180667392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17431891658415 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745067 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130886800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17645328991232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1359 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2508772286566400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64373943 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570029734904832 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2508772303605760 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚570029629819904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2495863055114240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263Ï e Wl43/ _wandbruntime‚14169  _runtime‚14169ÂÜÝ3Š 3loss/SRqqt2/assignment_loss‚0.42498863987239516 /loss/FRt2/detection_loss‚1.1602417560484235 /loss/FBt1/detection_loss‚0.4535006427042255 %loss/total_loss‚6.929923925473098 trainer/global_step‚16499 _step‚1360 2loss/SRqqt1/assignment_loss‚0.4477785608496133 4loss/FBt2/assignment_loss‚0.00018990116582010382 1loss/SRqqt2/detection_loss‚0.5419357513953278 0loss/FBt2/detection_loss‚0.32689506980439903 epoch‚41 0loss/FRt2/assignment_loss‚0.6942621074931722 2loss/FBt1/assignment_loss‚9.58081214561288e-07 1loss/SRqqt1/detection_loss‚0.8757656721085224 $ _timestamp‚1.784354511278043e+09 _runtime‚14169.971346218 /loss/FRt1/detection_loss‚0.5135223491316344 0loss/FRt1/assignment_loss‚1.49084251681835010?œ‘3 _runtime‚14169.971346218 /loss/FRt1/detection_loss‚0.5135223491316344 0loss/FRt1/assignment_loss‚1.4908425168183501 3loss/SRqqt2/assignment_loss‚0.42498863987239516 /loss/FRt2/detection_loss‚1.1602417560484235 /loss/FBt1/detection_loss‚0.4535006427042255 %loss/total_loss‚6.929923925473098 trainer/global_step‚16499 _step‚1360 2loss/SRqqt1/assignment_loss‚0.4477785608496133 4loss/FBt2/assignment_loss‚0.00018990116582010382 1loss/SRqqt2/detection_loss‚0.5419357513953278 0loss/FBt2/detection_loss‚0.32689506980439903 epoch‚41 0loss/FRt2/assignment_loss‚0.6942621074931722 2loss/FBt1/assignment_loss‚9.58081214561288e-07 1loss/SRqqt1/detection_loss‚0.8757656721085224 $ _timestamp‚1.784354511278043e+09Ð  %u“’3‚(:ˆ Ô­ìÒºô†” gpu.0.gpu‚34.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½º•'“3‚(:† Ô­ìÒÔ‘û• disk.dm-0.in‚22946.89453125# proc.memory.rssMB‚ 2742.70703125 cpu‚26.200168696144974( disk.nvme0c0n1.out‚32984.44091796875 disk.sda2.in‚ 0.0078125" disk.nvme0n1.in‚22946.89453125& disk.nvme0n1.out‚32984.44091796875 disk.rbd1.out‚ 45.4111328125' proc.memory.availableMB‚ 434.3046875) disk./.usagePercent‚39.30175004907153 disk.drbd2165.in‚0# disk.dm-0.out‚32984.44091796875$ disk./.usageGB‚585.6244773864746 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 866.00390625 disk.sda.in‚ 18.55078125 network.recv‚ 3.777009e+06 disk.sda3.in‚ 18.54296875$ memory_percent‚97.34921455383301) proc.memory.percent‚16.74015522003174 disk.sda1.in‚0 disk.dm-1.in‚ 18.54296875 disk.sda.out‚ 889.48828125 disk.sda2.out‚ 23.484375 disk.sda3.out‚ 866.00390625 disk.rbd1.in‚ 2.01953125$ disk.nvme0c0n1.in‚22946.89453125 network.sent‚ 2.6389914e+07rEŸnž”3‚(:“ ã­ìÒÿØÑ“ gpu.0.gpu‚72.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÖ‚•3‚(:ý ã­ìÒ‹šõ• disk.rbd1.out‚ 45.4111328125( proc.memory.availableMB‚ 434.46484375 disk.rbd0.in‚0 disk.sda3.in‚ 83.8671875 disk.sda3.out‚ 882.4765625 disk.sda2.out‚ 23.5390625& disk.nvme0n1.out‚32987.76416015625$ memory_percent‚97.34823703765869 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚22946.89453125) proc.memory.percent‚16.74015522003174 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚32987.76416015625 disk.sda.in‚83.875 network.sent‚ 2.6453365e+07 disk.sda1.out‚0 disk.dm-1.in‚ 83.8671875 disk.dm-1.out‚ 882.4765625 disk.sda.out‚ 906.015625" disk.nvme0n1.in‚22946.89453125 cpu‚26.199623687194876) disk./.usagePercent‚39.30175900933522# proc.memory.rssMB‚ 2742.70703125 network.recv‚ 3.781726e+06 disk.rbd1.in‚ 2.01953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚22946.89453125# disk.dm-0.out‚32987.76416015625 disk.sda2.in‚ 0.0078125$ disk./.usageGB‚585.6246109008789Óã@K“–3‚(:ˆ ò­ìÒ½ûΓ gpu.0.gpu‚58.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚208.998' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.7996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘k—3‚(:† ò­ìÒÂŽ—– disk.drbd2165.out‚0 disk.dm-0.in‚22946.89453125 disk.sda.out‚ 1166.25390625" disk.nvme0n1.in‚22946.89453125 proc.cpu.threads‚17$ disk./.usageGB‚585.6164779663086' disk.nvme0c0n1.out‚32993.2041015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 85.41796875 disk.dm-1.out‚ 1142.71484375 network.recv‚ 3.78504e+06) disk./.usagePercent‚39.30121320127222 disk.sda3.in‚ 85.41796875 disk.rbd1.out‚ 45.4736328125( proc.memory.availableMB‚ 434.38671875) proc.memory.percent‚16.74015522003174$ disk.nvme0c0n1.in‚22946.89453125 disk.sda3.out‚ 1142.71484375" disk.dm-0.out‚32993.2041015625 disk.sda2.in‚ 0.0078125 disk.sda2.out‚ 23.5390625 disk.rbd1.in‚ 2.01953125 disk.drbd2165.in‚0 cpu‚26.199131232077292 network.sent‚ 2.6458379e+07 disk.sda1.out‚0 disk.sda.in‚ 85.42578125% disk.nvme0n1.out‚32993.2041015625# memory_percent‚97.3487138748169# proc.memory.rssMB‚ 2742.70703125Æ €24˜3/ _wandbruntime‚14209  _runtime‚14209ŸhÕQÈ™3 _step‚1361 % lr-AdamW/pg1‚3.34839377025284e-05 % lr-AdamW/pg2‚3.34839377025284e-05 trainer/global_step‚16549 % _timestamp‚1.7843545509118767e+09 _runtime‚14209.606366907áX‰Íš3Ç % lr-AdamW/pg1‚3.34839377025284e-05 % lr-AdamW/pg2‚3.34839377025284e-05 trainer/global_step‚16549 % _timestamp‚1.7843545509118767e+09 _runtime‚14209.606366907 _step‚1361Ñ 2²·4›3/ _wandbruntime‚14209  _runtime‚14209”ìÃì/œ3€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131260768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784354550912684e+09 _runtime‚14209.606453831 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2533495318939136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17508204881408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2503042078035968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153303669 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2515987114057728 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2515987097018368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131260768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2520523234577567 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2533494960560640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2533495318939136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35572638 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚16549 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284574414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚554022328608768 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2520523592166817 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153303669 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17481497091489 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17507863542272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17481156541599 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35572367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571717936655872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚"„õ×17507863542272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131260766 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64561589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2533494960560640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571717827369472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153313646 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131260766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28988948 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚554022309996544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17695608047104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2503042095075328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1362 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17507863542272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131260766 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64561589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2533494960560640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571717827369472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153313646 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131260766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28988948 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚554022309996544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17695608047104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2503042095075328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1362 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17481497091489 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17507863542272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17481156541599 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ò ¢OR.4ž3/ _wandbruntime‚14210  _runtime‚14210;lšä'Ÿ3‹N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2503188605747200 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64565350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2520671113892523 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17482508145323 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284581119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28990696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2516134338048000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153322712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554055738755072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131268400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2533643572602880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571752281979904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚14210.219821451 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64565071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2533643214217216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17508876169216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17696543224832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2503188588707840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153312721 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17482167595321 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2516134338048000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2503188605747200 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64565350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2520671113892523 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17482508145323 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284581119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28990696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2516134338048000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153322712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554055738755072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131268400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2533643572602880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571752281979904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚14210.219821451 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64565071 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2533643214217216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.ö‰œ` on_train_batch_end/allocated_bytes.small_pool.freed‚17508876169216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17696543224832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2503188588707840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153312721 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17482167595321 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2516134338048000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2540907707544064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2540908065922560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17746692333568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153768098 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17746780910592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17558900383232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2527899742247325 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2510367628668928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843545908896308e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2523349148499968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573440547445760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2540907707544064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2540908065922560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17746692333568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153768098 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17746780910592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17558900383232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2527899742247325 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2510367628668928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843545908896308e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2523349148499968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573440547445760 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64755936 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285417171 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131650000 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17533124632231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29077943 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2528047263973031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17559913017344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555727195293696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17559913017344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427164 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2523496406568960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2523496406568960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35677993 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2523496389529600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843545915022564e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285417171 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2541056319586304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285427164 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153767173 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17747725304320¡Òÿ'²3N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285417171 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2541056319586304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285427164 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153767173 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17747725304320 trainer/global_step‚16599 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131649998 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64755936 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285417171 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131650000 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17533124632231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29077943 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2528047263973031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17559913017344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555727195293696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17559913017344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427164 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2523496406568960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2523496406568960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35677993 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2523496389529600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843545915022564e+09× Ð¨Ž4³3/ _wandbruntime‚14250  _runtime‚14250²ÓÍ ´3‡ 0loss/FRt1/assignment_loss‚1.4401083484508888 3loss/FBt1/assignment_loss‚0.0002245333715181531 .loss/FRt2/detection_loss‚1.204120468832498 %loss/total_loss‚7.198635190231958 trainer/global_step‚16599 _runtime‚14250.199366872 0loss/FRt2/assignment_loss‚0.6788334470778588 2loss/SRqqt1/assignment_loss‚0.4295881197331311 3loss/FBt2/assignment_loss‚0.0036937272621120477 1loss/SRqqt2/detection_loss‚0.5665696028228128 epoch‚41 $ _timestamp‚1.784354591506051e+09 /loss/FRt1/detection_loss‚0.5252686198564882 1loss/SRqqt1/detection_loss‚0.9254541755366055 /loss/FBt1/detection_loss‚0.5525275342502133 /loss/FBt2/detection_loss‚0.4485760366672762 _step‚1368 2loss/SRqqt2/assignment_loss‚0.4236705763705547ÿ"¢µ3Œ 2loss/SRqqt2/assignment_loss‚0.4236705763705547 0loss/FRt1/assignment_loss‚1.4401083484508888 3loss/FBt1/assignment_loss‚0.0002245333715181531 .loss/FRt2/detection_loss‚1.204120468832498 %loss/total_loss‚7.198635190231958 trainer/global_step‚16599 _runtime‚14250.199366872 0loss/FRt2/assignment_loss‚0.6788334470778588 2loss/SRqqt1/assignment_loss‚0.4295881197331311 3loss/FBt2/assignment_loss‚0.0036937272621120477 1loss/SRqqt2/detection_loss‚0.5665696028228128 epoch‚41 $ _timestamp‚1.784354591506051e+09 /loss/FRt1/detection_loss‚0.5252686198564882 1loss/SRqqt1/detection_loss‚0.9254541755366055 /loss/FBt1/detection_loss‚0.5525275342502133 /loss/FBt2/detection_loss‚0.4485760366672762 _step‚1368Ø ¸ÚÄ“¶3‚(:ˆ ®®ìÒÙ›• gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚212.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.4482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xe5·3‚(:ª ®®ìÒßð·– disk.sda1.in‚ 4.447265625# disk.dm-0.out‚33009.26708984375 disk.sda.in‚300.6142578125" disk.nvme0n1.in‚23031.62890625 disk.rbd0.in‚ 0.16015625$ memory_percent‚97.35045433044434 disk.sda1.out‚0 disk.sda.out‚ 1496.39453125 cpu‚26.197087554395637 network.sent‚ 2.6583983e+07 disk.dm-1.in‚ 150.1640625$ disk.nvme0c0n1.in‚23031.62890625 disk.sda3.in‚ 242.017578125 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0( disk./.usagePercent‚39.3017423688455 disk.dm-0.in‚ 23024.59375 proc.cpu.threads‚17 disk.sda2.in‚ 37.740234375& disk.nvme0n1.out‚33009.26708984375 disk.rbd1.out‚ 45.6611328125' proc.memory.availableMB‚ 434.1015625# proc.memory.rssMB‚ 2742.70703125) proc.memory.percent‚16.74015522003174 disk.drbd2165.out‚0 disk.sda3.out‚ 1326.40234375 disk.dm-1.out‚ 1326.40234375 disk.drbd2165.in‚ 2.25390625( disk.nvme0c0n1.out‚33009.26708984375 disk.sda2.out‚ 169.9921875 network.recv‚ 3.800683e+06$ disk./.usageGB‚585.6243629455566ó\ž¸3‚(:“ ½®ìÒ‘×Ì“ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AƒG÷1¹3‚(:¦ ½®ìÒ®ôñ• disk.sda3.in‚ 369.400390625 disk.rbd1.out‚ 45.6611328125 disk.rbd0.in‚ 0.16015625 disk.sda3.out‚ 1389.171875 disk.dm-1.in‚ 185.734375 disk.dm-1.out‚ 1389.171875$ disk.nvme0c0n1.in‚23038.62890625) proc.memory.percent‚16.74015522003174 disk.sda1.out‚0* disk./.usagePercent‚39.301752097131796 disk.drbd2165.in‚ 2.25390625# disk.dm-0.out‚33012.47705078125 disk.sda.in‚484.6298828125 network.sent‚ 2.6636541e+07 disk.rbd1.in‚ 2.1796875 disk.sda2.out‚ 170.3828125 network.recv‚ 3.805222e+06" disk.nvme0n1.in‚23038.62890625& disk.nvme0n1.out‚33012.47705078125 disk.drbd2165.out‚0 disk.sda.out‚ 1559.5546875( proc.memory.availableMB‚ 434.05078125# proc.memory.rssMB‚ 2742.70703125 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625( disk.nvme0c0n1.out‚33012.47705078125 disk.sda2.in‚ 73.833984375 disk.dm-0.in‚ 23024.59375$ memory_percent‚97.35076427459717 cpu‚26.19658602169798 proc.cpu.threads‚17$ disk./.usageGB‚585.6245079040527_Eþ'4º3/ _wandbruntime‚14289  _runtime‚14289?¿Ç»3Á _step‚1369 % lr-AdamW/pg1‚3.16177578745495e-05 % lr-AdamW/pg2‚3.16177578745495e-05 trainer/global_step‚16649 $ _timestamp‚1.784354630701105e+09 _runtime‚14289.394710893f—Ÿ ̼3Æ trainer/global_step‚16649 $ _timestamp‚1.784354630701105e+09 _runtime‚14289.394710893 _step‚1369 % lr-AdamW/pg1‚3.16177578745495e-05 % lr-AdamW/pg2‚3.16177578745495e-05Ù e•Õ<4½3/ _wandbruntime‚14289  _runtime‚14289´À|F(¾3P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2530711217020928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17797968493568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154212573 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17609595885056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚16649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132023968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575163259435008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557365204461568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1370 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29163448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35780023 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2535275534738583 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17609595885056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17609254545920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64943209 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚557365185849344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2535275892327833 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64943474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29163451 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35779761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286236539 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 % _timestamp‚1.7843546307022493e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2530711199981568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2548320812905984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575163154342912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132023966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17582730065305 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154212573 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2517693162262528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2548320812905984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2548320454527488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246518 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2530711217020928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17798054973440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154222550 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154222550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2530711199981568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17582389515415 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575163154342912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132023966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17582730065305 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154212573 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2517693162262528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2548320812905984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2548320454527488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246518 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2530711217020928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17798054973440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154222550 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154222550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2530711199981568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17582389515415 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35781772 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64946965 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2535423414053539 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17610267172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2548469066569728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132031598 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2535423056464177 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2517839672934400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2548468708184064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575197614904320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132031600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154221625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17610608519168 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583741119139 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154231616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2530858458050560 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154221625 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35782042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2517839655895040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273¯çõ 'Â3N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2548468708184064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575197614904320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132031600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154221625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17610608519168 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583741119139 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154231616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2530858458050560 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154221625 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35782042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2517839655895040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17799082574848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132031598 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚557398614607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29165193 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _step‚1371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286253223 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35781772 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64946965 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2535423414053539 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17610267172864 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2548469066569728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132031598 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2535423056464177 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2517839672934400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088Û =W‰®4Ã3/ _wandbruntime‚14290  _runtime‚14290hy6åÄ3‰ 0loss/FRt1/assignment_loss‚1.4744244433954632 2loss/FBt2/assignment_loss‚0.004299773536051241 /loss/FRt1/detection_loss‚0.4772089909685845 /loss/FRt2/detection_loss‚1.2744106589459903 /loss/FBt1/detection_loss‚0.4151576909353025 %loss/total_loss‚7.023812798308457 epoch‚41 trainer/global_step‚16649 3loss/SRqqt2/assignment_loss‚0.43046903720201474 4loss/FBt1/assignment_loss‚2.3149101406811348e-06 _step‚1372 2loss/SRqqt1/assignment_loss‚0.4077765823081721 0loss/SRqqt1/detection_loss‚0.784870306741479 $ _timestamp‚1.784354631318832e+09 0loss/FRt2/assignment_loss‚0.7553227662804569 1loss/SRqqt2/detection_loss‚0.5919024809609518 0loss/FBt2/detection_loss‚0.40796775212385095 _runtime‚14290.012143463;—&Å3Ž 0loss/FRt2/assignment_loss‚0.7553227662804569 1loss/SRqqt2/detection_loss‚0.5919024809609518 0loss/FBt2/detection_loss‚0.40796775212385095 _runtime‚14290.012143463 0loss/FRt1/assignment_loss‚1.4744244433954632 2loss/FBt2/assignment_loss‚0.004299773536051241 /loss/FRt1/detection_loss‚0.4772089909685845 /loss/FRt2/detection_loss‚1.2744106589459903 /loss/FBt1/detection_loss‚0.4151576909353025 %loss/total_loss‚7.023812798308457 epoch‚41 trainer/global_step‚16649 3loss/SRqqt2/assignment_loss‚0.43046903720201474 4loss/FBt1/assignment_loss‚2.3149101406811348e-06 _step‚1372 2loss/SRqqt1/assignment_loss‚0.4077765823081721 0loss/SRqqt1/detection_loss‚0.784870306741479 $ _timestamp‚1.784354631318832e+09Ü )°´ÉžÆ3‚(:“ Ì®ìÒ×Æç“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.746599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýî¯Ç3‚(:¤ Ì®ìÒÛ½Œ– disk.rbd1.out‚ 45.7236328125 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625' disk.nvme0c0n1.out‚33018.6904296875 network.recv‚ 3.808396e+06 disk.sda3.in‚ 415.427734375 disk.dm-1.in‚ 231.76171875 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625 network.sent‚ 2.6641695e+07 disk.sda1.in‚ 8.759765625! disk.nvme0c0n1.in‚ 23103.96875 disk.dm-0.in‚23089.93359375 disk.sda2.in‚ 82.544921875% disk.nvme0n1.out‚33018.6904296875 disk.sda1.out‚0 eE¢-disk.sda.in‚539.3681640625 disk.sda.out‚ 1598.5625( proc.memory.availableMB‚ 434.03515625 disk.rbd1.in‚ 2.1796875 cpu‚26.196065180772298$ disk./.usageGB‚585.6167869567871 disk.sda3.out‚ 1428.02734375 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1428.02734375 disk.nvme0n1.in‚ 23103.96875) proc.memory.percent‚16.74015522003174 proc.cpu.threads‚17$ memory_percent‚97.35085964202881) disk./.usagePercent‚39.30123393788249" disk.dm-0.out‚33018.6904296875# proc.memory.rssMB‚ 2742.70703125*ÈöžÈ3‚(:“ Û®ìÒšò¾“ gpu.0.gpu‚91.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.439400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ä×S2É3‚(:§ Û®ìÒóå•# proc.memory.rssMB‚ 2742.70703125 cpu‚26.195566986538335 network.recv‚ 3.812971e+06) disk./.usagePercent‚39.30174953705646 disk.sda.out‚ 1660.36328125 disk.dm-1.in‚ 231.80859375 proc.cpu.threads‚17 network.sent‚ 2.6704787e+07 disk.sda2.in‚ 82.544921875 disk.dm-0.in‚23089.93359375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚33023.67529296875) proc.memory.percent‚16.74015522003174$ disk./.usageGB‚585.6244697570801 disk.sda.in‚539.4150390625 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625 disk.nvme0n1.in‚ 23103.96875 disk.sda3.in‚ 415.474609375( proc.memory.availableMB‚ 434.03515625 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚33023.67529296875 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚33023.67529296875 disk.sda3.out‚ 1489.828125 disk.rbd1.out‚ 45.7236328125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 23103.96875 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.dm-1.out‚ 1489.828125$ memory_percent‚97.35085964202881¬âR‘Ê3‚(:† ê®ìÒ汈” gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—7Í!Ë3‚(:– ê®ìÒϛ͖ network.sent‚ 2.6709717e+07 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625 disk.sda1.out‚0 disk.dm-0.out‚ 33030.90625 disk.sda.in‚539.4150390625 disk.rbd1.out‚ 45.7861328125( proc.memory.availableMB‚ 434.04296875$ disk./.usageGB‚585.6244621276855 disk.dm-1.in‚ 231.80859375 disk.nvme0n1.in‚ 23103.96875 disk.rbd1.in‚ 2.1796875) proc.memory.percent‚16.74015522003174 disk.drbd2165.out‚0 disk.nvme0n1.out‚ 33030.90625 disk.sda3.in‚ 415.474609375 disk.rbd0.in‚ 0.16015625! disk.nvme0c0n1.in‚ 23103.96875 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1490.48828125$ memory_percent‚97.35081195831299# proc.memory.rssMB‚ 2742.70703125 proc.cpu.threads‚17 network.recv‚ 3.8155e+06) disk./.usagePercent‚39.30174902504139 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1490.48828125 disk.sda.out‚ 1661.0234375 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 33030.90625 disk.dm-0.in‚23089.93359375 cpu‚26.195078791181306þäjh4Ì3/ _wandbruntime‚14329  _runtime‚14329"ÊÍ3Ä _runtime‚14329.431129876 _step‚1373 & lr-AdamW/pg1‚3.070322459864017e-05 & lr-AdamW/pg2‚3.070322459864017e-05 trainer/global_step‚16699 % _timestamp‚1.7843546707375736e+09‹¸Œ¢ÏÎ3É _runtime‚14329.431129876 _step‚1373 & lr-AdamW/pg1‚3.070322459864017e-05 & lr-AdamW/pg2‚3.070322459864017e-05 trainer/global_step‚16699 % _timestamp‚1.7843546707375736e+09Ý öXí4Ï3/ _wandbruntime‚14329  _runtime‚14329D'sÔ(Ð3P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17659950047744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843546707387385e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132405568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚14329.432457172 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17659950047744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132405566 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154667025 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287082570 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542652042408341 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2525018695856128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132405568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559036642387968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65133768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287072591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154677002 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154677002 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154667025 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17633006002323 _step‚1374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2555733201510912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29250698 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576885826735616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17849287342592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29250701 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2525018678816768 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2538073251463168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2555733559889408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559036623775744 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2555733559889408Vრ(Ñ3†P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2555733559889408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559036623775744 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2555733559889408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17659950047744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843546707387385e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132405568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚14329.432457172 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17659950047744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132405566 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154667025 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287082570 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542652042408341 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2525018695856128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132405568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559036642387968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65133768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287072591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154677002 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154677002 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154667025 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17633006002323 _step‚1374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2555733201510912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29250698 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576885826735616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17849287342592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29250701 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2525018678816768 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2538073251463168Þ ¼"†4Ò3/ _wandbruntime‚14330  _runtime‚14330ß¹>½Ó3‹N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154686068 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2538220509532160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29252446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2538220492492800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2555881455167488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2542799206544685 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287099268 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132413200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132413198 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559070052534272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154676077 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154676077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17661304020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17660962674688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17850222704640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287089275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2538220509532160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132413198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559070071146496 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2555881813553152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2538220492492800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65137318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35884875 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35885143 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154686068 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17660962674688 OADeviceStatsMonitor.on_train_batch_end/allocatedPâ˪T_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65137589 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29252443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17634357606047 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2542799564134047 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17661304020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576920275238912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843546713519423e+09 _runtime‚14330.045729893 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17661304020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17660962674688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17850222704640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287089275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2538220509532160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132413198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559070071146496 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2555881813553152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2538220492492800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65137318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35884875 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35885143 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154686068 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17660962674688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65137589 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29252443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17634357606047 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2542799564134047 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17661304020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576920275238912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843546713519423e+09 _runtime‚14330.045729893 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29252446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2538220492492800ß †ò%4Õ3/ _wandbruntime‚14330  _runtime‚14330º(!JÖ3‰ 0loss/FRt1/assignment_loss‚1.4303484852982578 0loss/FRt2/assignment_loss‚0.9077047818865855 3loss/SRqqt1/assignment_loss‚0.42764788909535656 4loss/FBt1/assignment_loss‚2.6370476657541186e-07 /loss/FBt2/detection_loss‚0.3230506676039222 trainer/global_step‚16699 _runtime‚14330.048547164 .loss/FRt1/detection_loss‚0.575309867705638 0loss/FBt1/detection_loss‚0.29928691014929704 epoch‚41 2loss/SRqqt2/assignment_loss‚0.4064154686083148 1loss/SRqqt1/detection_loss‚0.8829538899910997 1loss/SRqqt2/detection_loss‚0.6395956802379611 % _timestamp‚1.7843546713552341e+09 _step‚1376 2loss/FBt2/assignment_loss‚0.001764933577945726 /loss/FRt2/detection_loss‚1.1425742356250153 $loss/total_loss‚7.03665307348416^N5×3Ž .loss/FRt1/detection_loss‚0.575309867705638 0loss/FBt1/detection_loss‚0.29928691014929704 epoch‚41 2loss/SRqqt2/assignment_loss‚0.4064154686083148 1loss/SRqqt1/detection_loss‚0.8829538899910997 1loss/SRqqt2/detection_loss‚0.6395956802379611 % _timestamp‚1.7843546713552341e+09 _step‚1376 2loss/FBt2/assignment_loss‚0.001764933577945726 /loss/FRt2/detection_loss‚1.1425742356250153 $loss/total_loss‚7.03665307348416 0loss/FRt1/assignment_loss‚1.4303484852982578 0loss/FRt2/assignment_loss‚0.9077047818865855 3loss/SRqqt1/assignment_loss‚0.42764788909535656 4loss/FBt1/assignment_loss‚2.6370476657541186e-07 /loss/FBt2/detection_loss‚0.3230506676039222 trainer/global_step‚16699 _runtime‚14330.048547164à lÑQ9Ø3‚(:„ ù®ìÒßö” gpu.0.gpu‚3.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 å½3Ù3‚(:¨ ù®ìÒ¢°ƒ— disk.sda1.in‚ 8.759765625) disk./.usagePercent‚39.30174057679276$ disk./.usageGB‚585.6243362426758 disk.sda1.out‚0 disk.dm-1.in‚ 231.80859375 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.35231399536133 disk.dm-1.out‚ 1490.75390625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 23103.96875 disk.nvme0n1.in‚ 23103.96875 disk.sda3.in‚ 415.474609375& proc.memory.availableMB‚ 433.796875 disk.dm-0.in‚23089.93359375# proc.memory.rssMB‚ 2742.70703125# disk.dm-0.out‚33036.33935546875 disk.sda.in‚539.4150390625 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚33036.33935546875) proc.memory.percent‚16.74015522003174( disk.nvme0c0n1.out‚33036.33935546875 disk.sda.out‚ 1661.2890625 disk.sda3.out‚ 1490.75390625 disk.rbd1.out‚ 45.7861328125 cpu‚26.194577708089444 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625 proc.cpu.threads‚17 network.sent‚ 2.6714034e+07 network.recv‚ 3.819156e+06H%žwžÚ3‚(:“ ˆ¯ìÒ¨ ø” gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚211.133' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.226600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ ð|5Û3‚(:ª ˆ¯ìÒÄûç˜ disk.dm-1.in‚ 231.80859375 disk.sda.in‚539.4150390625 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23190.93359375 disk.sda2.in‚ 82.544921875 network.sent‚ 2.6777324e+07 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚33044.46533203125 disk.rbd1.out‚ 45.9111328125$ memory_percent‚97.35221862792969$ proc.memory.availableMB‚433.8125 cpu‚26.194089863793074 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1661.5390625 disk.sda3.out‚ 1491.00390625 disk.rbd1.in‚ 2.1796875( disk./.usagePercent‚39.3017439048907( disk.nvme0c0n1.out‚33044.46533203125# disk.dm-0.out‚33044.46533203125" disk.nvme0n1.in‚23190.93359375) proc.memory.percent‚16.74015522003174 proc.cpu.threads‚17 network.recv‚ 3.823769e+06$ disk./.usageGB‚585.6243858337402# proc.memory.rssMB‚ 2742.70703125 disk.dm-1.out‚ 1491.00390625 disk.dm-0.in‚ 23176.8984375 disk.sda3.in‚ 415.474609375³SE4Ü3/ _wandbruntime‚14369  _runtime‚14369öÓÌÝ3Æ _runtime‚14369.411987614 _step‚1377 ' lr-AdamW/pg1‚2.9801140998417546e-05 ' lr-AdamW/pg2‚2.9801140998417546e-05 trainer/global_step‚16749 % _timestamp‚1.7843547107184277e+09žš³›ÑÞ3Ë % _timestamp‚1.7843547107184277e+09 _runtime‚14369.411987614 _step‚1377 ' lr-AdamW/pg1‚2.9801140998417546e-05 ' lr-AdamW/pg2‚2.9801140998417546e-05 trainer/global_step‚16749á õ¡aÀ4ß3/ _wandbruntime‚14369  _runtime‚14369ó?ùW(à3ÿO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532344229449728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65324142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578608454067200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17710645549568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29337951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚14369.41349972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17900489330688 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155131454 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2545435319984128 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2545435319984128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17900392365056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2545435302944768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17710645549568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2563145948494336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132787166 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155131454 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155121477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29337948 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155121477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35986447 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1378 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132787168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚560708061702144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2550028192488849 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2563146306872832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578608569645056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2550027834899599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65324398 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17683622489231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784354710719523e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287918622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17710986888704 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2545435302944768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17710986888704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35986194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17683963039121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2550027834899599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65324398 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17683622489231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784354710719523e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287918622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17710986888704 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2545435302944768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17710986888704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35986194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17683963039121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532344229449728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65324142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578608454067200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_stJÌCØÓart/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17710645549568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29337951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚14369.41349972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17900489330688 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155131454 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2545435319984128 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2545435319984128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17900392365056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2545435302944768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17710645549568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2563145948494336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132787166 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155131454 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155121477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29337948 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155121477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35986447 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1378â ¹›'“â3‚(:ˆ —¯ìÒ•ýÍ“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èy£}7ã3‚(:¬ —¯ìÒªŒ’– disk.sda1.in‚ 8.759765625' disk.nvme0c0n1.out‚33050.1904296875 disk.sda2.in‚ 82.544921875% disk.nvme0n1.out‚33050.1904296875 disk.sda3.in‚ 415.474609375 disk.rbd1.out‚ 45.9111328125 disk.dm-1.in‚ 231.80859375 disk.drbd2165.out‚0 disk.sda.in‚539.4150390625 cpu‚26.193567955512158 network.recv‚ 3.827649e+06 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚23190.93359375" disk.dm-0.out‚33050.1904296875 disk.sda.out‚ 1662.1015625" disk.nvme0n1.in‚23190.93359375 disk.sda3.out‚ 1491.56640625( proc.memory.availableMB‚ 434.41796875 network.sent‚ 2.6795211e+07) disk./.usagePercent‚39.30174672097358 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.34852313995361# proc.memory.rssMB‚ 2742.70703125 disk.dm-1.out‚ 1491.56640625 disk.dm-0.in‚ 23176.8984375 proc.cpu.threads‚17$ disk./.usageGB‚585.6244277954102 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625) proc.memory.percent‚16.74015522003174UÊQ4ä3/ _wandbruntime‚14370  _runtime‚14370ZËs'å3ŠN REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29339693 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35988453 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2532490740121600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578642912474112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17711658176512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287925327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711658176512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65328149 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65327870 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17684974092955 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578643019656192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2545582543974400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132794800 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2563294560536576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560741490460672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2545582561013760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17901422013440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132794798 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17684633542953 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132794798 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2563294560536576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560741509072896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚14370.028487223 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711658176512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65328149 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65327870 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17684974092955 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578643019656192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2545582543974400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132794800 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2563294560536576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560741490460672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2545582561013760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17901422013440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132794798 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17684633542953 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132794798 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2563294560536576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560741509072896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚14370.028487223 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578642912474112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17711658176512ã èGøÔ4ç3/ _wandbruntime‚14370  _runtime‚14370á÷qè3‰ 3loss/FBt1/assignment_loss‚5.455013514913768e-05 3loss/FBt2/assignment_loss‚0.0003330127334363787 /loss/FRt2/detection_loss‚1.1151566557028785 /loss/FBt2/detection_loss‚0.3409076221510606 %loss/total_loss‚6.677780755258096 0loss/FRt1/assignment_loss‚1.3970135970711073 0loss/FRt2/assignment_loss‚0.6726122721036275 3loss/SRqqt1/assignment_loss‚0.36964443977548034 /loss/FRt1/detection_loss‚0.5609072259377175 1loss/SRqqt1/detection_loss‚0.9294403128893259 _step‚1380 2loss/SRqqt2/assignment_loss‚0.3544012026085217 /loss/FBt1/detection_loss‚0.3649066450448255 trainer/global_step‚16749 _runtime‚14370.030743493 1loss/SRqqt2/detection_loss‚0.5724032191049663 epoch‚41 $ _timestamp‚1.784354711337443e+09Ð ¶òé3Ž 3loss/FBt1/assignment_loss‚5.455013514913768e-05 3loss/FBt2/assignment_loss‚0.0003330127334363787 /loss/FRt2/detection_loss‚1.1151566557028785 /loss/FBt2/detection_loss‚0.3409076221510606 %loss/total_loss‚6.677780755258096 0loss/FRt1/assignment_loss‚1.3970135970711073 0loss/FRt2/assignment_loss‚0.6726122721036275 3loss/SRqqt1/assignment_loss‚0.36964443977548034 /loss/FRt1/detection_loss‚0.5609072259377175 1loss/SRqqt1/detection_loss‚0.9294403128893259 _step‚1380 2loss/SRqqt2/assignment_loss‚0.3544012026085217 /loss/FBt1/detection_loss‚0.3649066450448255 trainer/global_step‚16749 _runtime‚14370.030743493 1loss/SRqqt2/detection_loss‚0.5724032191049663 epoch‚41 $ _timestamp‚1.784354711337443e+09ä ‚´)žê3‚(:“ ¦¯ìÒå†Ì“ gpu.0.gpu‚46.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cë‹ï8ë3‚(:­ ¦¯ìÒÀ£ß• disk.sda3.in‚ 415.474609375 disk.sda.out‚ 1662.37109375) proc.memory.percent‚16.74015522003174 cpu‚26.19378427643625 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1491.8359375 disk.rbd1.in‚ 2.1796875" disk.nvme0n1.in‚23190.93359375# proc.memory.rssMB‚ 2742.70703125( proc.memory.availableMB‚ 434.38671875 disk.sda1.out‚0 disk.dm-1.out‚ 1491.8359375 disk.dm-0.in‚ 23176.8984375 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 46.0361328125 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚23190.93359375* disk./.usagePercent‚39.301761569410566$ disk./.usageGB‚585.6246490478516 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.80859375 network.sent‚ 2.6834881e+07 network.recv‚ 3.831991e+06# memory_percent‚97.3487138748169 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625( disk.nvme0c0n1.out‚33057.63427734375# disk.dm-0.out‚33057.63427734375 disk.sda.in‚539.4150390625& disk.nvme0n1.out‚33057.634277343757ÝÈ“ì3‚(:ˆ µ¯ìÒÜé¿“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©©E4í3‚(:© µ¯ìÒ›÷Ý•$ disk.nvme0c0n1.in‚23190.93359375 disk.dm-0.in‚ 23176.8984375" disk.nvme0n1.in‚23190.93359375 disk.rbd1.out‚ 46.0361328125* disk./.usagePercent‚39.353488915733955 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.80859375 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚33091.162109375 disk.sda2.out‚ 170.53515625$ disk.nvme0n1.out‚33091.162109375$ disk./.usageGB‚586.3954238891602! disk.dm-0.out‚33091.162109375 disk.sda.out‚ 1662.88671875$ memory_percent‚97.34880924224854 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625 disk.sda.in‚539.4150390625 disk.rbd1.in‚ 2.1796875 cpu‚26.193912061198326( proc.memory.availableMB‚ 434.37109375 network.recv‚ 3.843852e+06 disk.dm-1.out‚ 1492.3515625 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1492.3515625# proc.memory.rssMB‚ 2742.70703125 disk.sda3.in‚ 415.474609375 network.sent‚ 2.6891262e+07) proc.memory.percent‚16.74015522003174F#c4î3/ _wandbruntime‚14409  _runtime‚14409¾ïk+Ëï3Å ' lr-AdamW/pg2‚2.8911565013025734e-05 trainer/global_step‚16799 % _timestamp‚1.7843547510637066e+09 _runtime‚14409.75730454 _step‚1381 ' lr-AdamW/pg1‚2.8911565013025734e-05¦ô¦Ðð3Ê ' lr-AdamW/pg1‚2.8911565013025734e-05 ' lr-AdamW/pg2‚2.8911565013025734e-05 trainer/global_step‚16799 % _timestamp‚1.7843547510637066e+09 _runtime‚14409.75730454 _step‚1381å ZqƒŠ4ñ3/ _wandbruntime‚14409  _runtime‚14409¸”•ò3P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2552797354426368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17951560202240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2552797371465728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133168766 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155575929 _step‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2539669763043328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754674 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2539669746003968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562379518240768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚14409.758546305 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17761682390528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2570559053856256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29425201 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17761341051392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2552797371465728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562379499628544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133168766 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133168768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29425198 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155585906 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17761341051392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17761682390528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2570558695477760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17951650876416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17734238976139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2570558695477760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17734579526029 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2557404342569357 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2570559053856256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580331059830784 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2557403984980107 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚16799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155585906 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65515069 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580331169117184‹Íå[ (ó3†P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2570558695477760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17734579526029 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2557404342569357 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2570559053856256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580331059830784 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2557403984980107 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚16799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155585906 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65515069 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580331169117184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2552797354426368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17951560202240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2552797371465728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133168766 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155575929 _step‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2539669763043328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754674 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2539669746003968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562379518240768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚14409.758546305 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17761682390528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2570559053856256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29425201 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17761341051392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2552797371465728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562379499628544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133168766 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133168768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29425198 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155585906 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17761341051392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17761682390528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2570558695477760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17951650876416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17734238976139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169æ Ê'c4ô3/ _wandbruntime‚14410  _runtime‚144107Z'õ3ŠN QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17952668484096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761379 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2557551506705701 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2557551864295063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17952584108544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17735250029861 trainer/global_step‚16799 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155584981 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2570706949134336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29426946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29426943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17735590579863 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2539816273715200 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155594972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133176398 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1383 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133176398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2539816256675840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761379 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133176400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65518527 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2570707307520000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2552944595456000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771372 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580365512495616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2552944595456000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580365615483392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133176400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155594972 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17762353678336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0_ü¶` ö3N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2557551864295063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17952584108544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17735250029861 trainer/global_step‚16799 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155584981 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2570706949134336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29426946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29426943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17735590579863 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2539816273715200 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155594972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133176398 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1383 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133176398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADef1 âúviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2539816256675840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761379 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133176400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65518527 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2570707307520000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2552944595456000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771372 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580365512495616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2552944595456000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580365615483392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133176400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155594972 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17762353678336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17952668484096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761379 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2557551506705701 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ç >5Q–4÷3/ _wandbruntime‚14410  _runtime‚14410“£©ø3ˆ 1loss/SRqqt1/detection_loss‚0.7557582295744412 /loss/FBt1/detection_loss‚0.5748116939955961 _step‚1384 0loss/FRt1/assignment_loss‚1.4821940533871656 0loss/FRt2/assignment_loss‚0.6881271194276356 3loss/SRqqt1/assignment_loss‚0.33219793511476936 % _timestamp‚1.7843547516814377e+09 2loss/FBt1/assignment_loss‚0.012801212051675195 /loss/FRt1/detection_loss‚0.5467633584188323 1loss/SRqqt2/detection_loss‚0.5018916882738169 0loss/FBt2/detection_loss‚0.42092424597568523 %loss/total_loss‚6.950772354166494 epoch‚41 _runtime‚14410.374723621 2loss/SRqqt2/assignment_loss‚0.4231922845492255 1loss/FBt2/assignment_loss‚0.00513968294304074 /loss/FRt2/detection_loss‚1.2069708504546102 trainer/global_step‚167990¼Rêù3 epoch‚41 _runtime‚14410.374723621 2loss/SRqqt2/assignment_loss‚0.4231922845492255 1loss/FBt2/assignment_loss‚0.00513968294304074 /loss/FRt2/detection_loss‚1.2069708504546102 trainer/global_step‚16799 1loss/SRqqt1/detection_loss‚0.7557582295744412 /loss/FBt1/detection_loss‚0.5748116939955961 _step‚1384 0loss/FRt1/assignment_loss‚1.4821940533871656 0loss/FRt2/assignment_loss‚0.6881271194276356 3loss/SRqqt1/assignment_loss‚0.33219793511476936 % _timestamp‚1.7843547516814377e+09 2loss/FBt1/assignment_loss‚0.012801212051675195 /loss/FRt1/detection_loss‚0.5467633584188323 1loss/SRqqt2/detection_loss‚0.5018916882738169 0loss/FBt2/detection_loss‚0.42092424597568523 %loss/total_loss‚6.950772354166494è hz=ú3‚(:’ įìÒϦ” gpu.0.gpu‚31.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.105' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.621000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`*Ê+û3‚(:  įìÒ˜•– disk.sda3.out‚ 1492.8125# proc.memory.rssMB‚ 2748.62109375 disk.dm-0.in‚23271.38671875 disk.sda.out‚ 1663.34765625 disk.sda3.in‚ 415.474609375 disk.rbd1.out‚ 46.0986328125 disk.sda1.out‚0 disk.dm-1.in‚ 231.80859375* proc.memory.percent‚16.776251792907715$ memory_percent‚97.38376140594482( proc.memory.availableMB‚ 428.64453125 disk.sda2.out‚ 170.53515625" disk.nvme0c0n1.in‚ 23285.421875 disk.sda.in‚539.4150390625 disk.nvme0n1.in‚ 23285.421875 disk.rbd1.in‚ 2.1796875 cpu‚26.195197567088563 disk.drbd2165.out‚0! disk.dm-0.out‚33097.685546875 proc.cpu.threads‚17) disk./.usagePercent‚39.35169353489633 disk.drbd2165.in‚ 2.25390625 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.rbd0.out‚0 disk.dm-1.out‚ 1492.8125 network.sent‚ 2.6896375e+07 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚33097.685546875$ disk./.usageGB‚586.3686714172363 network.recv‚ 3.847425e+06& disk.nvme0c0n1.out‚33097.685546875ÐÛ]Í4ü3/ _wandbruntime‚14428  _runtime‚14428k¨TÚ…=ý3ÿz 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23051400482654572 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3014325797557831 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2874010503292084 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_3‚0.9261944890022278 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7945270538330078 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6570343971252441 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09385818243026733 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 *jet/accuracy_1_of_1‚0.3489000201225281 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7185248136520386 +particle/specificity‚0.8995103240013123 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03782660514116287 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7275182604789734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23953711986541748 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7714766263961792 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.795087456703186 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6354181170463562 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3790574371814728 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6418355107307434 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2875271439552307 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7895639538764954 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2490941286087036 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2840224802494049 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29839378595352173 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36593300104141235 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24738268554210663 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8839324116706848 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037697382271289825 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6407731175422668 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.631660521030426 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21686366200447083 particle/accuracy_1_of_4‚1 _runtime‚14428.527563378 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _step‚1385 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5689501762390137 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10462687164545059 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333743155002594 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7678950428962708 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8191420435905457 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13002203404903412 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.780916690826416 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12574650347232819 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5552765727043152 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0981433168053627 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6435167193412781 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.38172027468681335 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2793269753456116 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2981019914150238 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34159356355667114 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6869389414787292 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_2_of_3‚0.5131664872169495 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843547698330116e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5017942190170288 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.73463374376297 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9486842155456543 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026549408212304115 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16140198707580566 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2400621473789215 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7174816131591797 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24986350536346436 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6966739296913147 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28765979409217834 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14315417408943176 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7960985898971558 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.784345805644989 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016057904809713364 +jet/accuracy_3_of_4‚0.04078979045152664 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5681696534156799 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023318497464060783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6562132835388184 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 )loss/val_total_loss‚7.697620970267454 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9410266876220703 'particle/f_score‚0.6061543226242065 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8255857825279236 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3188782334327698 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21609361469745636 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28765979409217834 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04860231280326843 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8786066174507141 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7643676996231079 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4922349154949188 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7881911396980286 (particle/accuracy‚0.8340502977371216 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8839324116706848 trainer/global_step‚16799 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14642226696014404 ,jet/accuracy_3_of_3‚0.038326092064380646 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23820260167121887 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016057904809713364 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03565774857997894 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023462386801838875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7371767163276672 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8442554473876953 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.688322126865387 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5738356113433838 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +jet/accuracy_2_of_2‚0.12816956639289856 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6955563426017761 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18553011119365692 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026777077466249466 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 jet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.720207691192627 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14457760751247406 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25120073556900024 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7522091269493103 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8442554473876953 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 jet/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.01670061983168125 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658298909664154 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6407731175422668 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7348374128341675 *jet/accuracy_2_of_4‚0.7744083404541016 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2427428811788559 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7785682678222656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6154257655143738 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22331498563289642 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7180148363113403 *jet/accuracy_1_of_4‚0.9806751012802124 +particle/sensitivity‚0.5946907997131348 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6743527054786682 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6730297803878784 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8554043769836426 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 epoch‚41 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6604681611061096 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7161862254142761 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05406659096479416 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5390989184379578 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚05Óbñþ3„{ ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2400621473789215 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7174816131591797 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24986350536346436 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6966739296913147 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28765979409217834 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14315417408943176 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7960985898971558 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.784345805644989 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016057904809713364 +jet/accuracy_3_of_4‚0.04078979045152664 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5681696534156799 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023318497464060783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6562132835388184 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 )loss/val_total_loss‚7.697620970267454 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9410266876220703 'particle/f_score‚0.6061543226242065 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8255857825279236 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3188782334327698 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21609361469745636 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28765979409217834 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04860231280326843 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8786066174507141 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7643676996231079 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4922349154949188 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7881911396980286 (particle/accuracy‚0.8340502977371216 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8839324116706848 trainer/global_step‚16799 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14642226696014404 ,jet/accuracy_3_of_3‚0.038326092064380646 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23820260167121887 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016057904809713364 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03565774857997894 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.023462386801838875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7371767163276672 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8442554473876953 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.688322126865387 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5738356113433838 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +jet/accuracy_2_of_2‚0.12816956639289856 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6955563426017761 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18553011119365692 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026777077466249466 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 jet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.720207691192627 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14457760751247406 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25120073556900024 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7522091269493103 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8442554473876953 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 jet/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.01670061983168125 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658298909664154 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6407731175422668 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7348374128341675 *jet/accuracy_2_of_4‚0.7744083404541016 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2427428811788559 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7785682678222656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6154257655143738 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22331498563289642 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7180148363113403 *jet/accuracy_1_of_4‚0.9806751012802124 +particle/sensitivity‚0.5946907997131348 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_pu~ûÓ ™!rity‚0.6743527054786682 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6730297803878784 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8554043769836426 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 epoch‚41 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6604681611061096 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7161862254142761 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05406659096479416 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5390989184379578 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23051400482654572 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3014325797557831 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2874010503292084 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_3‚0.9261944890022278 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7945270538330078 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6570343971252441 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09385818243026733 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 *jet/accuracy_1_of_1‚0.3489000201225281 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7185248136520386 +particle/specificity‚0.8995103240013123 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03782660514116287 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7275182604789734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23953711986541748 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7714766263961792 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.795087456703186 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6354181170463562 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3790574371814728 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6418355107307434 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2875271439552307 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7895639538764954 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2490941286087036 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2840224802494049 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29839378595352173 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36593300104141235 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24738268554210663 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8839324116706848 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037697382271289825 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6407731175422668 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.631660521030426 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21686366200447083 particle/accuracy_1_of_4‚1 _runtime‚14428.527563378 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _step‚1385 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5689501762390137 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10462687164545059 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333743155002594 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7678950428962708 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8191420435905457 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13002203404903412 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.780916690826416 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12574650347232819 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5552765727043152 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0981433168053627 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6435167193412781 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.38172027468681335 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2793269753456116 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2981019914150238 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34159356355667114 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6869389414787292 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_2_of_3‚0.5131664872169495 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843547698330116e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5017942190170288 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.73463374376297 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9486842155456543 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026549408212304115 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16140198707580566é Eks£“ÿ3‚(:ˆ Ó¯ìÒí»ñ” gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚245.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÚÑŒ2€4‚(:§ Ó¯ìÒõ× disk.nvme0n1.in‚ 23285.421875 proc.cpu.threads‚17 network.sent‚ 2.6959469e+07 disk.dm-1.out‚ 1493.2890625 disk.sda3.out‚ 1493.2890625" disk.nvme0c0n1.in‚ 23285.421875$ memory_percent‚97.52614498138428 cpu‚26.19848270492027 network.recv‚ 3.85196e+06) disk./.usagePercent‚39.30290720312629 disk.rbd0.in‚ 0.16015625 disk.rbd1.in‚ 2.1796875& disk.nvme0n1.out‚33825.31201171875 disk.rbd1.out‚ 46.0986328125 disk.sda1.out‚0 disk.dm-1.in‚ 231.80859375 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚23271.38671875 disk.sda.in‚539.4150390625 disk.sda.out‚ 1663.82421875 disk.sda2.in‚ 82.544921875( proc.memory.availableMB‚ 405.31640625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚33825.31201171875 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 415.474609375" proc.memory.rssMB‚ 2771.9609375( proc.memory.percent‚16.9187068939209$ disk./.usageGB‚585.6417198181152 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚33825.31201171875ÆJ¿64‚(:’ â¯ìÒùöž” gpu.0.gpu‚83.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éX«{4‚4‚(:© â¯ìÒؽ¥–$ memory_percent‚97.52671718597412 disk.rbd0.in‚ 0.16015625" disk.nvme0c0n1.in‚ 23285.421875 disk.dm-0.in‚23271.38671875 disk.sda.in‚539.4150390625 disk.nvme0n1.in‚ 23285.421875( proc.memory.availableMB‚ 405.22265625$ disk./.usageGB‚585.6417274475098( disk.nvme0c0n1.out‚33859.95751953125" proc.memory.rssMB‚ 2771.9609375 network.sent‚ 2.7017098e+07 disk.dm-1.out‚ 1493.6328125 disk.sda3.out‚ 1493.6328125) disk./.usagePercent‚39.30290771514136 disk.sda.out‚ 1664.16796875 disk.sda2.in‚ 82.544921875 cpu‚26.198005238038323 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 415.474609375( proc.memory.percent‚16.9187068939209 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0# disk.dm-0.out‚33859.95751953125& disk.nvme0n1.out‚33859.95751953125 disk.rbd1.out‚ 46.1611328125 proc.cpu.threads‚17 network.recv‚ 3.855984e+06 disk.sda1.out‚0 disk.dm-1.in‚ 231.80859375 disk.rbd1.in‚ 2.1796875…›ã‰‘ƒ4‚(:† ñ¯ìÒÜþ¾“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñä‘.„4‚(:£ ñ¯ìÒ î´• disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625' proc.memory.availableMB‚ 405.2421875 disk.sda1.out‚0 disk.sda2.in‚ 82.544921875$ memory_percent‚97.52659797668457# disk./.usageGB‚585.615650177002 disk.dm-1.out‚ 1494.0859375" disk.nvme0c0n1.in‚ 23285.421875 cpu‚26.197510224035835 network.sent‚ 2.7022078e+07 network.recv‚ 3.85947e+06% disk.nvme0n1.out‚33866.8857421875 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.sda.in‚539.4150390625 disk.sda.out‚ 1664.62109375 disk.sda1.in‚ 8.759765625 disk.sda2.out‚ 170.53515625) disk./.usagePercent‚39.30115764763729 disk.nvme0n1.in‚ 23285.421875 disk.sda3.out‚ 1494.0859375 disk.rbd1.out‚ 46.1611328125' disk.nvme0c0n1.out‚33866.8857421875" proc.memory.rssMB‚ 2771.9609375( proc.memory.percent‚16.9187068939209 disk.sda3.in‚ 415.474609375 disk.dm-0.in‚23271.38671875" disk.dm-0.out‚33866.8857421875 proc.cpu.threads‚17 disk.dm-1.in‚ 231.80859375‘+ÿG4…4/ _wandbruntime‚14467  _runtime‚14467¸LÎóʆ4Ä % _timestamp‚1.7843548091053457e+09 _runtime‚14467.798921686 _step‚1386 & lr-AdamW/pg1‚2.803455377826792e-05 & lr-AdamW/pg2‚2.803455377826792e-05 trainer/global_step‚16849æöiâχ4É % _timestamp‚1.7843548091053457e+09 _runtime‚14467.798921686 _step‚1386 & lr-AdamW/pg1‚2.803455377826792e-05 & lr-AdamW/pg2‚2.803455377826792e-05 trainer/global_step‚16849ê ™šô4ˆ4/ _wandbruntime‚14467  _runtime‚144679Æ’(‰4P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2579918742317056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17830111097856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17829769758720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771735 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65739307 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843548091064541e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156100703 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17830111097856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133671034 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2566706875589174 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17802575188276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133671032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2562088972558336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29514971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156110680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781714 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582087793178624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚564066868267008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2579919100695552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18021025809408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17829769758720 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2562088972558336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚564066849654784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2548903959851008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1387 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17802915738166 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156110680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582087894076416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36224607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269Ý¡ñ¯"Š4†P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843548091064541e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156100703 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17830111097856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133671034 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2566706875589174 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17802575188276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133671032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2562088972558336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29514971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156110680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781714 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582087793178624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚564066868267008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2579919100695552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18021025809408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17829769758720 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2562088972558336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚564066849654784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2548903959851008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1387 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17802915738166 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156110680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582087894076416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36224607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2579918742317056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17830111097856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17829769758720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771735 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65739307 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983ë Z­4‹4/ _wandbruntime‚14468  _runtime‚144680}Q+'Œ4ˆN OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133678664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133678664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2562236213587968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2580067354359296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17803926792000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2562236230627328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36226362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582122355364352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2562236213587968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2580066995973632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564100278413312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29516713 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289788419 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18021978157568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156109755 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18022058338816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚16849 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65743075 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29516716 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2549050453483520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1388 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133678666 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562236230627328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788419 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156119746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784354809719771e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2566854039725518 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17830782385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2566854397314880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36226622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582122256570880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17830782385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65743338 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2580067354359296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17831123731968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17803586241998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798412 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798412 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156119746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2549050470522880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17831123731968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚14468.41352604q:MÐ'4N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133678666 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2566854039725518 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17830782385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2566854397314880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36226622 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582122256570880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17830782385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65743338 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2580067354359296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17831123731968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17803586241998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798412 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798412 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156119746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2549050470522880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17831123731968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚14468.41352604 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133678664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133678664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2562236213587968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2580067354359296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17803926792000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2562236230627328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36226362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582122355364352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2562236213587968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2580066995973632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564100278413312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29516713 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289788419 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18021978157568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156109755 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18022058338816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚16849 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65743075 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29516716 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2549050453483520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1388 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133678666 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562236230627328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788419 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156119746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784354809719771e+09 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134052634 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17880465260544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17880806599680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2587331847678976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2587331489300480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156555155 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29602221 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18072509348864 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2556229476405248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17880465260544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65930135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583810815542272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 % _timestamp‚1.7843548491748688e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2574083025669682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134052634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2569451024039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156565132 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156565132 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2587331489300480 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2569451024039936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565738287581184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17853191675184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134052632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290607787 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583810718838784 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565738306193408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088? (œ4…P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2587331847678976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2587331489300480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156555155 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29602221 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18072509348864 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2556229476405248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17880465260544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65930135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583810815542272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 % _timestamp‚1.7843548491748688e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2574083025669682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134052634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2569451024039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156565132 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156565132 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2587331489300480 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2569451024039936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565738287581184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17853191675184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134052632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290607787 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583810718838784 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565738306193408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2574082668080432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134052632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29602218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36327917 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134052634 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17880465260544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17880806599680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269ï íFù×44/ _wandbruntime‚14508  _runtime‚14508E•;`'ž4‹N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29603963 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156564207 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚1392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17854202728906 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583845168402944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18073542730240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65933955 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29603966 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134060266 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17881819233792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843548497865927e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156574198 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290624471 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134060264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2574230547395388 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134060264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17881477887488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134060266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚16899 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2587479742957056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2556375987077120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17854543278908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565771716339712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2569598282108928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583845277682176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156574198 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36330265 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2569598265069568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17881477887488 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2587480101342720 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2574230189806026 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65934231 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2556376004116480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2587480101342720 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2569598265069568ÿ+o=µŸ4N QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17881477887488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134060266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚16899 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2587479742957056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2556375987077120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17854543278908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565771716339712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2569598282108928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583845277682176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156574198 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36330265 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2569598265069568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17881477887488 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2587480101342720 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2574230189806026 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65934231 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2556376004116480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2587480101342720 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2569598265069568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29603963 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMoni#]Ua tor.on_train_batch_end/active.small_pool.freed‚ 156564207 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚1392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17854202728906 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583845168402944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18073542730240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65933955 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29603966 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134060266 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17881819233792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843548497865927e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156574198 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290624471 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134060264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2574230547395388 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134060264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594744236283904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585533568779264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17931160762368 trainer/global_step‚16949 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2576813075521536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚1395 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291443839 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134434234 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66121966 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2563555027038208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134434234 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594744594662400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594744236283904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚585533667579904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134434232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2576813092560896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567409725507584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453818 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157009607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29689468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2576813075521536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154cÖ (¬4…P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚1395 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291443839 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134434234 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66121966 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2563555027038208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134434234 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594744594662400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594744236283904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚585533667579904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134434232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2576813092560896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567409725507584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453818 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157009607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29689468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2576813075521536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17931502101504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594744236283904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585533568779264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17931160762368 trainer/global_step‚16949 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2576813075521536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ó ä(Ñ4­4/ _wandbruntime‚14548  _runtime‚14548ð€ì&®4ŠN P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157018659 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17932173389312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2576960333590528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16949 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2581606697475896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17905159765816 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2576960316551168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36434561 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2563701537710080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66125515 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904819215814 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_;\ñÒüend/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2576960316551168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567443172878336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2563701520670720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66125777 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36434302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17932173389312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17932514735616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29691213 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157028650 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2594892848326144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2594892489940480 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2576960333590528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1396 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157028650 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2594892848326144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17932514735616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134441864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134441866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134441866 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚567443154266112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362iLÖ'¯4N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16949 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2581606697475896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17905159765816 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2576960316551168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36434561 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2563701537710080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66125515 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904819215814 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2576960316551168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567443172878336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2563701520670720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66125777 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36434302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17932173389312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17932514735616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29691213 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157028650 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2594892848326144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2594892489940480 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2576960333590528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1396 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157028650 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2594892848326144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17932514735616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134441864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134441866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134441866 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚567443154266112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157018659 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17932173389312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2576960333590528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ô Ÿ¡¹w4°4/ _wandbruntime‚14548  _runtime‚14548œC~‚ ±4‡ 3loss/SRqqt2/assignment_loss‚0.44999559481256146 3loss/FBt2/assignment_loss‚0.0005553627773779345 1loss/SRqqt2/detection_loss‚0.5276202693446025 /loss/FBt1/detection_loss‚0.5788850094986628 _runtime‚14548.30136191 _step‚1397 0loss/FRt1/assignment_loss‚1.5518820361380317 /loss/FRt1/detection_loss‚0.5533666356931309 0loss/SRqqt1/detection_loss‚0.855228764345936 /loss/FBt2/detection_loss‚0.4212755820184201 trainer/global_step‚16949 0loss/FRt2/assignment_loss‚0.6989001412934894 3loss/FBt1/assignment_loss‚0.0024119692983031182 %loss/total_loss‚7.224752373011622 epoch‚42 2loss/SRqqt1/assignment_loss‚0.4370350269233401 /loss/FRt2/detection_loss‚1.1475959808677672 $ _timestamp‚1.784354889608052e+09 ’1ñ²4Œ /loss/FBt2/detection_loss‚0.4212755820184201 trainer/global_step‚16949 0loss/FRt2/assignment_loss‚0.6989001412934894 3loss/FBt1/assignment_loss‚0.0024119692983031182 %loss/total_loss‚7.224752373011622 epoch‚42 2loss/SRqqt1/assignment_loss‚0.4370350269233401 /loss/FRt2/detection_loss‚1.1475959808677672 $ _timestamp‚1.784354889608052e+09 3loss/SRqqt2/assignment_loss‚0.44999559481256146 3loss/FBt2/assignment_loss‚0.0005553627773779345 1loss/SRqqt2/detection_loss‚0.5276202693446025 /loss/FBt1/detection_loss‚0.5788850094986628 _runtime‚14548.30136191 _step‚1397 0loss/FRt1/assignment_loss‚1.5518820361380317 /loss/FRt1/detection_loss‚0.5533666356931309 0loss/SRqqt1/detection_loss‚0.855228764345936õ ÿ{–ž³4‚(:“ Ë°ìÒã gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ïÇ-´4‚(:¢ Ë°ì҇阖) disk./.usagePercent‚39.30035890413046 disk.sda3.out‚ 1496.203125 disk.rbd1.out‚ 46.3486328125 disk.dm-1.in‚ 231.80859375' proc.memory.availableMB‚ 405.1015625 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 23850.8359375& disk.nvme0c0n1.out‚33892.111328125 disk.sda.out‚ 1666.73828125$ disk.nvme0n1.out‚33892.111328125 proc.cpu.threads‚17 network.sent‚ 2.7172688e+07$ disk./.usageGB‚585.6037483215332 disk.sda3.in‚ 415.474609375" proc.memory.rssMB‚ 2771.9609375 network.recv‚ 3.889276e+06 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚23836.80078125 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 23850.8359375 disk.sda1.in‚ 8.759765625 disk.sda2.out‚ 170.53515625$ memory_percent‚97.52745628356934( proc.memory.percent‚16.9187068939209 disk.rbd1.in‚ 2.1796875 cpu‚26.194384524772612! disk.dm-0.out‚33892.111328125 disk.sda.in‚539.4150390625 disk.dm-1.out‚ 1496.203125)œµÄµ4‚(:’ Ú°ìÒ ­¾“ gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚217.897' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚43.57939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤$î0¶4‚(:¥ Ú°ìÒ¶œÓ•* disk./.usagePercent‚39.300355832040054 disk.drbd2165.out‚0" disk.dm-0.out‚33895.2646484375 disk.rbd1.out‚ 46.4111328125 disk.rbd0.out‚0 disk.dm-1.out‚ 1496.609375! disk.nvme0n1.in‚ 23850.8359375 disk.sda3.out‚ 1496.609375 disk.rbd0.in‚ 0.16015625# disk.nvme0c0n1.in‚ 23850.8359375 disk.dm-0.in‚23836.80078125 network.recv‚ 3.893341e+06' disk.nvme0c0n1.out‚33895.2646484375" proc.memory.rssMB‚ 2771.9609375 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1667.14453125% disk.nvme0n1.out‚33895.2646484375 proc.cpu.threads‚17$ memory_percent‚97.52793312072754 network.sent‚ 2.7235741e+07 disk.sda1.out‚0 disk.sda3.in‚ 415.474609375 disk.rbd1.in‚ 2.1796875 cpu‚26.193912689708323 disk.sda1.in‚ 8.759765625 disk.sda.in‚539.4150390625 disk.sda2.in‚ 82.544921875' proc.memory.availableMB‚ 405.0234375( proc.memory.percent‚16.9187068939209# disk./.usageGB‚585.603702545166 disk.dm-1.in‚ 231.80859375 disk.sda2.out‚ 170.53515625…nÿž·4‚(:“ é°ìÒïÓ“ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AŽL†7¸4‚(:¬ é°ìÒ‡Þá• disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 415.474609375 network.sent‚ 2.7240896e+07# disk.nvme0c0n1.in‚ 23850.8359375( disk.nvme0c0n1.out‚33901.22119140625 disk.sda.in‚539.4150390625 disk.rbd1.in‚ 2.1796875( proc.memory.availableMB‚ 405.01171875" proc.memory.rssMB‚ 2771.9609375 cpu‚26.193422229411624 disk.drbd2165.out‚0( proc.memory.percent‚16.9187068939209 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625# disk.dm-0.out‚33901.22119140625$ memory_percent‚97.52800464630127 network.recv‚ 3.89715e+06 disk.rbd0.out‚0 disk.sda3.out‚ 1496.98828125 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 231.80859375 disk.dm-0.in‚23836.80078125 disk.sda.out‚ 1667.5234375 disk.rbd1.out‚ 46.4111328125* disk./.usagePercent‚39.300350711889365$ disk./.usageGB‚585.6036262512207 disk.dm-1.out‚ 1496.98828125 disk.sda1.in‚ 8.759765625! disk.nvme0n1.in‚ 23850.8359375& disk.nvme0n1.out‚33901.22119140625 disk.sda1.out‚0b¹8Ï4¹4/ _wandbruntime‚14587  _runtime‚14587–A"ë̺4Æ ' lr-AdamW/pg1‚2.5479467809018483e-05 ' lr-AdamW/pg2‚2.5479467809018483e-05 trainer/global_step‚16999 % _timestamp‚1.7843549289628615e+09 _runtime‚14587.656434514 _step‚1398XûñÑ»4Ë % _timestamp‚1.7843549289628615e+09 _runtime‚14587.656434514 _step‚1398 ' lr-AdamW/pg1‚2.5479467809018483e-05 ' lr-AdamW/pg2‚2.5479467809018483e-05 trainer/global_step‚16999ö ?ˆ›4¼4/ _wandbruntime‚14587  _runtime‚14587 Í‹º&½4€P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157474036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587256282370048 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17981856264192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66312650 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2602156983267328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134815834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36535929 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569081163433984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2602157341645824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2584175144042496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134815832 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157474036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserv¥ âLed_bytes.all.current‚ 24406654976 _step‚1399 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2584175144042496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2584175127003136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17982197603328ŸëFð (¾4…P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134815832 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157474036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1399 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2584175144042496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2584175127003136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17982197603328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157474036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587256282370048 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17981856264192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66312650 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2602156983267328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134815834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36535929 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569081163433984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2602157341645824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2584175144042496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587290728230912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17955776252724 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292296575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157483102 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚569114610804736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843549295768957e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2584322368032768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292296575 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2602305595309568 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2584322385072128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134823464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36537690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18176136038400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134823466 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157473111 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚14588.270649315 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17983210237440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571027071303680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2584322368032768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571027054264320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66316153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888«©Î'Á4N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587290728230912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17955776252724 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292296575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157483102 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚569114610804736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843549295768957e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2584322368032768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292296575 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2602305595309568 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2584322385072128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134823464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36537690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18176136038400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134823466 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157473111 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚14588.270649315 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17983210237440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571027071303680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2584322368032768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571027054264320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66316153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588978963441664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36639183 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157928488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135197432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2609569730250752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135197434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570752601360384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2609570088629248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18032893105152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157928488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157918511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36638928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18032551766016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2591537178484736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18032551766016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚1403 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2591537178484736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66503154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66502896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18005381685798 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135197434 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157918511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570752619972608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293115943 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588979068533760 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2596211475911206;£x (Ð4…P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293125922 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 $ _timestamp‚1.784354969011781e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2578206077186048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18005041135908 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29863971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2596211118321956 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2609569730250752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2609570088629248 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2591537195524096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2591537195524096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚17049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29863968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚14627.705496019 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2591537178484736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18032551766016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚1403 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2591537178484736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66503154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66502896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18005381685798 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135197434 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157918511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570752619972608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293115943 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588979068533760 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2596211475911206 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2578206094225408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135197432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18032893105152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18226362081280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588978963441664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36639183 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157928488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135197432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2609569730250752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135197434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570752601360384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2609570088629248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18032893105152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157928488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157918511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36638928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18032551766016 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2596358640047550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157927563 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589013525434368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18006052189630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚17049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2609717983907328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66506985 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29865713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚570786048731136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18033905739264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36640992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2591684436553728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2591684419514368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66506705 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843549696250336e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132627 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157937554 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18227394424832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29865716 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36641269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157927563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2578352587857920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2591684436553728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142620 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135205066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135205064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18033564392960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18227476703232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMo‡‘Âî@ nitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2578352604897280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157937554 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589013424543744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18033905739264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2596358997636912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2609718342292992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1404 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609718342292992 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2591684419514368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135205064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18006392739632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚ 14628.3187851ñÈ—Z'Ó4ŽN WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18033564392960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18227476703232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2578352604897280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157937554 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589013424543744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18033905739264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2596358997636912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2609718342292992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1404 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609718342292992 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2591684419514368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135205064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18006392739632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚ 14628.3187851 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2596358640047550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157927563 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589013525434368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18006052189630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚17049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2609717983907328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66506985 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29865713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚570786048731136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18033905739264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36640992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2591684436553728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2591684419514368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66506705 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843549696250336e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132627 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157937554 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18227394424832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29865716 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36641269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157927563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2578352587857920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2591684436553728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142620 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135205066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135205064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ü )Å4Ô4/ _wandbruntime‚14628  _runtime‚14628°Ù&¡Õ4‰ % _timestamp‚1.7843549696281643e+09 1loss/SRqqt1/assignment_loss‚0.378498062701439 epoch‚42 _step‚1405 3loss/FBt2/assignment_loss‚3.259367392505861e-06 1loss/SRqqt1/detection_loss‚0.8390977032768943 1loss/SRqqt2/detection_loss‚0.5706168650512903 0loss/FBt2/detection_loss‚0.38583746695364807 %loss/total_loss‚6.878724526343657 trainer/global_step‚17049 0loss/FRt1/assignment_loss‚1.4537645718374432 0loss/FRt2/assignment_loss‚0.7144034990044528 2loss/SRqqt2/assignment_loss‚0.4472045356958406 3loss/FBt1/assignment_loss‚4.811198265268028e-06 /loss/FRt2/detection_loss‚1.2371855878326212 /loss/FBt1/detection_loss‚0.2826875567213089 _runtime‚14628.321697789 /loss/FRt1/detection_loss‚0.5694206067030607i™ qÖ4Ž 3loss/FBt2/assignment_loss‚3.259367392505861e-06 1loss/SRqqt1/detection_loss‚0.8390977032768943 1loss/SRqqt2/detection_loss‚0.5706168650512903 0loss/FBt2/detection_loss‚0.38583746695364807 %loss/total_loss‚6.878724526343657 trainer/global_step‚17049 0loss/FRt1/assignment_loss‚1.4537645718374432 0loss/FRt2/assignment_loss‚0.7144034990044528 2loss/SRqqt2/assignment_loss‚0.4472045356958406 3loss/FBt1/assignment_loss‚4.811198265268028e-06 /loss/FRt2/detection_loss‚1.2371855878326212 /loss/FBt1/detection_loss‚0.2826875567213089 _runtime‚14628.321697789 /loss/FRt1/detection_loss‚0.5694206067030607 % _timestamp‚1.7843549696281643e+09 1loss/SRqqt1/assignment_loss‚0.378498062701439 epoch‚42 _step‚1405ý j™Ö¢“×4‚(:ˆ ¥±ìÒÊ«Æ“ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚245.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q™†*Ø4‚(:Ÿ ¥±ìÒÅ€à– disk.sda3.in‚ 415.474609375 disk.drbd2165.in‚ 2.25390625 proc.cpu.threads‚17 network.sent‚ 2.7379264e+07 disk.sda.out‚ 1668.828125 disk.sda2.in‚ 82.544921875 disk.rbd0.out‚0 cpu‚26.191451458732246$ disk./.usageGB‚585.6036186218262 disk.nvme0n1.in‚ 24151.234375" proc.memory.rssMB‚ 2771.9609375" proc.memory.availableMB‚404.25 network.recv‚ 3.912634e+06( disk./.usagePercent‚39.3003501998743 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.dm-1.out‚ 1498.29296875" disk.nvme0c0n1.in‚ 24151.234375" disk.dm-0.out‚33919.9716796875 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1498.29296875 disk.rbd1.out‚ 46.5361328125 disk.rbd0.in‚ 0.16015625 disk.drbd2165.out‚0$ memory_percent‚97.53265380859375( proc.memory.percent‚16.9187068939209 disk.sda1.in‚ 8.759765625' disk.nvme0c0n1.out‚33919.9716796875 disk.sda.in‚539.4150390625% disk.nvme0n1.out‚33919.9716796875 disk.dm-0.in‚24137.19921875 disk.dm-1.in‚ 231.808593754Ð^“Ù4‚(:ˆ ´±ìÒ×⬓ gpu.0.gpu‚99.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÙ"3Ú4‚(:¨ ´±ìÒÀµ°— disk.sda1.out‚0 disk.dm-0.in‚24137.19921875 disk.nvme0n1.in‚ 24151.234375" proc.memory.rssMB‚ 2771.9609375 disk.rbd1.in‚ 2.1796875" disk.nvme0c0n1.in‚ 24151.234375' proc.memory.availableMB‚ 404.4921875 cpu‚26.19096411026975 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1498.61328125) disk./.usagePercent‚39.30034840782156 disk.dm-1.out‚ 1498.61328125 disk.drbd2165.out‚0 disk.sda.out‚ 1669.1484375$ disk./.usageGB‚585.6035919189453 disk.rbd0.out‚0 disk.sda.in‚539.4150390625& disk.nvme0n1.out‚33922.73291015625 disk.sda3.in‚ 415.474609375 proc.cpu.threads‚17 network.recv‚ 3.915901e+06 disk.dm-1.in‚ 231.80859375( disk.nvme0c0n1.out‚33922.73291015625# disk.dm-0.out‚33922.73291015625 disk.sda2.out‚ 170.53515625( proc.memory.percent‚16.9187068939209 disk.rbd0.in‚ 0.16015625 disk.rbd1.out‚ 46.5986328125$ memory_percent‚97.53117561340332 network.sent‚ 2.7382984e+07uŠžÝ4Û4/ _wandbruntime‚14667  _runtime‚14667Y€ÌÜ4Æ % _timestamp‚1.7843550088525608e+09 _runtime‚14667.546074787 _step‚1406 ' lr-AdamW/pg1‚2.3839911925227446e-05 ' lr-AdamW/pg2‚2.3839911925227446e-05 trainer/global_step‚17099bcÉÑÝ4Ë % _timestamp‚1.7843550088525608e+09 _runtime‚14667.546074787 _step‚1406 ' lr-AdamW/pg1‚2.3839911925227446e-05 ' lr-AdamW/pg2‚2.3839911925227446e-05 trainer/global_step‚17099þ (»¹‰4Þ4/ _wandbruntime‚14667  _runtime‚14667ª-(ß4P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18083247267840 _runtime‚14667.547367577 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158382940 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158372963 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2603587268402464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2598899229966336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158382940 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158372963 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2616982835612672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66693920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2585531627819008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585531610779648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2598899229966336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36742699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293961974 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843550088536477e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2598899247005696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590701726802432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572424057899008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083247267840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚1407 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135579032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572424039286784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18277576132096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66693641 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29951221 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135579034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36742423 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2603587625991714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2616982477234176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2616982835612672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269œÐ¼Ì]à4†P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135579034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2616982477234176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135579032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18277668903424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18055657622816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083588606976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2598899247005696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18083247267840 _runtime‚14667.547367577 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158382940 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158372963 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2603587268402464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2598899229966336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158382940 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158372963 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2616982835612672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66693920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2585531627819008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585531610779648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2598899229966336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36742699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatst¥«'¯Monitor.on_train_batch_start/allocation.all.allocated‚ 293961974 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843550088536477e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2598899247005696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590701726802432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572424057899008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083247267840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚1407 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135579032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572424039286784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18277576132096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66693641 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29951221 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135579034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36742423 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2603587625991714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2616982477234176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2616982835612672 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66697751 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2603735147717420 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66697492 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18084601241088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135586664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18084259894784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29952966 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2603734790128058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293968679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚572457468045312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590736175814656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚590736072826880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18056668676538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36744785 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585678138490880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158392006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29952963 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18278689157120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2617131089276416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2585678121451520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18084601241088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2617131089276416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2585678121451520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18084601241088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66697751 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2603735147717420 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66697492 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18084601241088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135586664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18084259894784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29952966 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2603734790128058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293968679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚572457468045312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590736175814656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚590736072826880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18056668676538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36744785 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585678138490880€ IAÃ4ä4/ _wandbruntime‚14668  _runtime‚14668}…¼¼ å4† 4loss/FBt1/assignment_loss‚0.00025925609928825516 -loss/FRt2/detection_loss‚1.18722005937825 1loss/SRqqt2/detection_loss‚0.5687505981306777 /loss/FBt1/detection_loss‚0.3602374432704112 3loss/SRqqt1/assignment_loss‚0.40055246776185843 1loss/FBt2/assignment_loss‚0.00300064548222265 /loss/FBt2/detection_loss‚0.3655224673090035 trainer/global_step‚17099 % _timestamp‚1.7843550094707594e+09 _runtime‚14668.164066828 0loss/FRt1/assignment_loss‚1.4880933120672928 1loss/SRqqt1/detection_loss‚0.8122863079568603 $loss/total_loss‚6.97440972672002 epoch‚42 _step‚1409 /loss/FRt1/detection_loss‚0.5235257976737471 0loss/FRt2/assignment_loss‚0.8256149695033119 2loss/SRqqt2/assignment_loss‚0.43934640208709571³%æ4‹ 3loss/SRqqt1/assignment_loss‚0.40055246776185843 1loss/FBt2/assignment_loss‚0.00300064548222265 /loss/FBt2/detection_loss‚0.3655224673090035 trainer/global_step‚17099 % _timestamp‚1.7843550094707594e+09 _runtime‚14668.164066828 0loss/FRt1/assignment_loss‚1.4880933120672928 1loss/SRqqt1/detection_loss‚0.8122863079568603 $loss/total_loss‚6.97440972672002 epoch‚42 _step‚1409 /loss/FRt1/detection_loss‚0.5235257976737471 0loss/FRt2/assignment_loss‚0.8256149695033119 2loss/SRqqt2/assignment_loss‚0.4393464020870957 4loss/FBt1/assignment_loss‚0.00025925609928825516 -loss/FRt2/detection_loss‚1.18722005937825 1loss/SRqqt2/detection_loss‚0.5687505981306777 /loss/FBt1/detection_loss‚0.3602374432704112 À‡Ñä“ç4‚(:ˆ ñìÒš–Ê“ gpu.0.gpu‚84.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m´[c.è4‚(:£ ñìÒîðŒ– disk.sda1.out‚0 disk.dm-1.out‚ 1499.0078125 disk.rbd1.out‚ 46.5986328125" proc.memory.rssMB‚ 2771.9609375 cpu‚26.190484803328943 network.sent‚ 2.7387493e+07 disk.rbd0.in‚ 0.16015625 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 24448.3203125 disk.sda3.out‚ 1499.0078125( proc.memory.percent‚16.9187068939209 disk.sda.in‚539.4150390625$ disk.nvme0n1.out‚33926.955078125 proc.cpu.threads‚17 network.recv‚ 3.91929e+06# disk.nvme0c0n1.in‚ 24448.3203125! disk.dm-0.out‚33926.955078125 disk.sda3.in‚ 415.474609375' proc.memory.availableMB‚ 404.4921875 disk.rbd0.out‚0 disk.sda.out‚ 1669.54296875 disk.dm-0.in‚24434.28515625) disk./.usagePercent‚39.30034891983662$ disk./.usageGB‚585.6035995483398 disk.sda1.in‚ 8.759765625 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.53117561340332 disk.dm-1.in‚ 231.80859375 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚33926.955078125︣`“é4‚(:ˆ Ò±ìÒ¼¢Ý“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä|¨(ê4‚(: Ò±ì҇趖& disk.nvme0n1.out‚33933.54931640625% proc.memory.availableMB‚ 404.65625 network.sent‚ 2.7449681e+07 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 24448.3203125 disk.dm-0.in‚24434.28515625 cpu‚26.18996656112646 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 24448.3203125 disk.rbd0.out‚0# disk.dm-0.out‚33933.54931640625 disk.sda.in‚539.4150390625 disk.sda3.in‚ 415.474609375 disk.sda3.out‚1500.625# memory_percent‚97.5301742553711 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.dm-1.in‚ 231.80859375 disk.dm-1.out‚1500.625( disk.nvme0c0n1.out‚33933.54931640625 disk.rbd1.out‚ 46.6611328125# disk./.usageGB‚585.603214263916 disk.sda.out‚ 1671.16015625 disk.sda2.out‚ 170.53515625 disk.rbd1.in‚ 2.1796875( proc.memory.percent‚16.9187068939209) disk./.usagePercent‚39.30032306307567 disk.sda2.in‚ 82.544921875 network.recv‚ 3.922875e+06" proc.memory.rssMB‚ 2771.9609375Nþ4ë4‚(:’ á±ìÒíȯ“ gpu.0.gpu‚18.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.900400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o~'4ì4‚(:© á±ìÒËÿ§•( proc.memory.availableMB‚ 404.88671875$ disk./.usageGB‚585.6030960083008( proc.memory.percent‚16.9187068939209 proc.cpu.threads‚17 disk.dm-0.in‚ 24434.3046875 disk.drbd2165.in‚ 2.25390625 cpu‚26.1894887159855( disk.nvme0c0n1.out‚33945.32177734375& disk.nvme0n1.out‚33945.32177734375 disk.sda1.out‚0 disk.dm-1.out‚ 1504.20703125# memory_percent‚97.5287675857544# disk.dm-0.out‚33945.32177734375 disk.sda3.in‚ 415.486328125 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 46.6611328125" proc.memory.rssMB‚ 2771.9609375 network.sent‚ 2.7454826e+07 network.recv‚ 3.926484e+06$ disk.nvme0c0n1.in‚24448.33984375" disk.nvme0n1.in‚24448.33984375) disk./.usagePercent‚39.30031512684211 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1504.20703125 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0 disk.sda.in‚539.4267578125 disk.sda.out‚ 1674.7421875 disk.sda2.in‚ 82.544921875 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 231.82031254ÖŒ4í4/ _wandbruntime‚14707  _runtime‚14707\ŠhÈî4 & lr-AdamW/pg1‚2.303944360322544e-05 & lr-AdamW/pg2‚2.303944360322544e-05 trainer/global_step‚17149 % _timestamp‚1.7843550488976443e+09 _runtime‚ 14707.5912393 _step‚1410oÿÕ-Íï4Ç _runtime‚ 14707.5912393 _step‚1410 & lr-AdamW/pg1‚2.303944360322544e-05 & lr-AdamW/pg2‚2.303944360322544e-05 trainer/global_step‚17149 % _timestamp‚1.7843550488976443e+09‚ Oe*4ð4/ _wandbruntime‚14707  _runtime‚14707ANêÝñ4€P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135960634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ¼Ãä)%QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135960634 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592424369394176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2592857161412608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1411 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2624395224217600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2592857144373248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843550488987608e+09 _runtime‚14707.59251599 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2624395224217600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038471 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135960632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158837392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66883821 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294788047 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18328776603136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2606261281447936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574095495825408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36845353 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2606261298487296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18133942769664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2610963418482972 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18106274109724 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158837392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18134284108800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158827415 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2624395582596096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36845626 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592424253816320 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2610963776072222 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18133942769664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496ïÔÚÜ (ò4…P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158837392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66883821 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294788047 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18328776603136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2606261281447936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574095495825408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36845353 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2606261298487296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18133942769664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2610963418482972 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18106274109724 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158837392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18134284108800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158827415 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2624395582596096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36845626 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592424253816320 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2610963776072222 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18133942769664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135960634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135960634 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592424369394176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2592857161412608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1411 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2624395224217600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2592857144373248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843550488987608e+09 _runtime‚14707.59251599 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2624395224217600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038471 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135960632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968ƒ å„s4ó4/ _wandbruntime‚14708  _runtime‚14708»Ü—'ô4‹N trainer/global_step‚17149 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2624543836259840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2606408539516928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66887566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843550495128975e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30040216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592458813808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18135296742912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606408522477568 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592458704528896 _runtime‚14708.206620595 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2606408539516928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135968266 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18134955396608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814724 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574128905971712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2593003655045120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18107625713448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66887855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2611111297797928 _step‚1412 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294814724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2593003672084480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158846458 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158836467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18135296742912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18134955396608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36847353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158846458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2624543836259840 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2624543477874176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2611110940208566 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158836467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135968264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18329889224192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18329798557184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438íÛ¶d õ4N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135968264 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30040213 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18107285163446 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135968266 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2624543836259840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2606408539516928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66887566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843550495128975e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30040216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592458813808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18135296742912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2370294937Nœx²6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606408522477568 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592458704528896 _runtime‚14708.206620595 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2606408539516928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135968266 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18134955396608 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814724 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574128905971712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2593003655045120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18107625713448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66887855 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2611111297797928 _step‚1412 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804731 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294814724 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2593003672084480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158846458 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158836467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18135296742912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18134955396608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36847353 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158846458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2624543836259840 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2624543477874176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2611110940208566 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158836467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135968264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18329889224192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18329798557184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚17149 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2631807971201024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30125721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2631808329579520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67074759 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚1415 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2600182677966848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2613623332929536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295634078 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18184979610624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136342232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575766933751808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594146993228800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36949041 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594146890233856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624099 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2618339568563480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚14747.457300168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159291844 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2613623349968896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159281867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2613623349968896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18184638271488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101w3§"„5…P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136342234 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136342234 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 $ _timestamp‚1.784355088760002e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159281867 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18184979610624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18184638271488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136342232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594146890233856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624099 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2618339568563480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚14747.457300168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159291844 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2613623349968896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159281867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2613623349968896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18184638271488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18380059476992 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2618339926152730 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18157231146522 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575766915139584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18156890596632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2631807971201024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30125721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2631808329579520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67074759 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚1415 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2600182677966848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2613623332929536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295634078 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18184979610624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136342232 QñäíIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575766933751808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594146993228800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36949041 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2613770573959168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18185992244736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18158242200356 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2600329205678080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159290919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36951369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136349864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2618487090289074 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30127463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843550893737814e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594181445927424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18381083417088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2631956583243264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575800362510336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2600329188638720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚17199 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2613770573959168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67078835 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18185650898432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2618487447878436 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18157901650354 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159300910 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18185992244736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67078566 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚14748.067525682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650776 _step‚1416¥Ä÷Ÿ'‡5N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18185992244736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67078566 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚14748.067525682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650776 _step‚1416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136349866 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2631956224857600 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2613770590998528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚594181334551040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575800343898112 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2613770573959168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18185992244736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18158242200356 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2600329205678080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159290919 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36951369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136349864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2618487090289074 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30127463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843550893737814e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594181445927424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18381083417088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2631956583243264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575800362510336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2600329188638720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚17199 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2613770573959168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67078835 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18185650898432 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2618487447878436 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18157901650354 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159300910ˆ 9>ÄÄ4ˆ5/ _wandbruntime‚14748  _runtime‚14748»‘ê‰5‰ %loss/total_loss‚6.928695644660991 .loss/FRt1/assignment_loss‚1.45044548054264 0loss/FRt2/assignment_loss‚0.7592202400719678 3loss/FBt1/assignment_loss‚2.992732616858898e-06 1loss/SRqqt2/detection_loss‚0.6482978129284814 0loss/FBt2/detection_loss‚0.35009754080686173 trainer/global_step‚17199 3loss/SRqqt1/assignment_loss‚0.43325809442428265 /loss/FRt1/detection_loss‚0.5766570468854331 /loss/FBt1/detection_loss‚0.2673506972802754 _runtime‚14748.070167327 epoch‚42 3loss/SRqqt2/assignment_loss‚0.43199403764515226 /loss/FRt2/detection_loss‚1.1753709212562105 1loss/SRqqt1/detection_loss‚0.8359924809484219 _step‚1417 3loss/FBt2/assignment_loss‚8.299138647498618e-06 $ _timestamp‚1.784355089376783e+09‘KåŠ5Ž 3loss/FBt1/assignment_loss‚2.992732616858898e-06 1loss/SRqqt2/detection_loss‚0.6482978129284814 0loss/FBt2/detection_loss‚0.35009754080686173 trainer/global_step‚17199 3loss/SRqqt1/assignment_loss‚0.43325809442428265 /loss/FRt1/detection_loss‚0.5766570468854331 /loss/FBt1/detection_loss‚0.2673506972802754 _runtime‚14748.070167327 epoch‚42 3loss/SRqqt2/assignment_loss‚0.43199403764515226 /loss/FRt2/detection_loss‚1.1753709212562105 1loss/SRqqt1/detection_loss‚0.8359924809484219 _step‚1417 3loss/FBt2/assignment_loss‚8.299138647498618e-06 $ _timestamp‚1.784355089376783e+09 %loss/total_loss‚6.928695644660991 .loss/FRt1/assignment_loss‚1.45044548054264 0loss/FRt2/assignment_loss‚0.7592202400719678‰ 9¯þH‘‹5‚(:† ²ìÒäæГ gpu.0.gpu‚60.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<©É*Œ5‚(:Ÿ ²ìÒ£¯®—# disk./.usageGB‚585.603328704834" disk.nvme0n1.out‚ 33967.8828125$ memory_percent‚97.61214256286621 network.recv‚ 3.940995e+06 disk.rbd1.out‚ 46.8486328125 disk.dm-1.out‚ 1505.67578125 disk.sda2.out‚ 170.53515625# proc.memory.rssMB‚ 2784.55078125 disk.dm-0.out‚ 33967.8828125 disk.sda.in‚539.4267578125' proc.memory.availableMB‚ 391.2265625 network.sent‚ 2.7534558e+07( disk./.usagePercent‚39.3003307433017 disk.sda3.out‚ 1505.67578125# disk.nvme0c0n1.in‚ 24598.5390625 disk.dm-0.in‚24584.50390625 disk.sda3.in‚ 415.486328125 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 33967.8828125 disk.sda2.in‚ 82.544921875* proc.memory.percent‚16.995549201965332 cpu‚26.190622592492044 disk.rbd1.in‚ 2.1796875 disk.sda.out‚ 1676.2109375 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 24598.5390625 disk.rbd0.out‚0 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625ˆVåZ45/ _wandbruntime‚14765  _runtime‚14765#ˆ¢Ä%Ž5ùz <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28524699807167053 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09744232147932053 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23933406174182892 particle/accuracy_3_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32151034474372864 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28375694155693054 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7161023020744324 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7320885062217712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7774187922477722 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.039887648075819016 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15315505862236023 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6728954911231995 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6381608247756958 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5691252946853638 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6428263783454895 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22736813127994537 particle/accuracy_2_of_4‚1 /particle/accuracy_4_of_4‚0.9426373243331909 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8649281859397888 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.535254180431366 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25622299313545227 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_3_of_3‚0.03835988789796829 'particle/f_score‚0.6058635711669922 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02035684697329998 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6576772928237915 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5512044429779053 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7188358902931213 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02296353690326214 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_5‚1 % _timestamp‚1.7843551066707873e+09 _runtime‚14765.365083231 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7760972380638123 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6168036460876465 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25477221608161926 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.5959442853927612 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2916331887245178 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2455238401889801 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2918345034122467 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30241066217422485 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.791361391544342 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807711005210876 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13852162659168243 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7177329063415527 *jet/accuracy_1_of_1‚0.3512841761112213 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7339175343513489 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0377722792327404 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7462345957756042 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.694096281948317 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023817570880055428 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03775270655751228 +jet/accuracy_2_of_2‚0.12640392780303955 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7240077257156372 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_1_of_2‚0.7902440428733826 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6611286401748657 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7127764821052551 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚42 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09789977967739105 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3026236891746521 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7677456140518188 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7144554853439331 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9968254566192627 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6306747198104858 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1699538975954Ð9s‡Z0558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24339736998081207 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6435505747795105 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7888989448547363 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2572992146015167 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.055872973054647446 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8759808540344238 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24092835187911987 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2921435236930847 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34390491247177124 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.566982090473175 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24516884982585907 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8323884606361389 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13999192416667938 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02035684697329998 jet/accuracy_4_of_5‚0 trainer/global_step‚17199 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36677446961402893 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02313581295311451 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6592488288879395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 #particle/accuracy‚ 0.83349609375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6849276423454285 jet/accuracy_3_of_5‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.731759786605835 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7926696538925171 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24321620166301727 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6329706311225891 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8759808540344238 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6837536692619324 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.302450567483902 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5120308995246887 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8286658525466919 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2921435236930847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.050458114594221115 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 +jet/accuracy_4_of_4‚0.01670061983168125 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3376815915107727 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.383461058139801 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17612631618976593 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7943505644798279 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15501584112644196 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.653121829032898 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10812331736087799 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.695926308631897 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6739528775215149 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22003750503063202 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.023589905351400375 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3802171051502228 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21916832029819489 +particle/specificity‚0.8984630107879639 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6428263783454895 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9806751012802124 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.694586455821991 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12087900191545486 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7688578367233276 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5752907395362854 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7743523716926575 _step‚1418 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9289266467094421 +jet/accuracy_3_of_4‚0.04078979045152664 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7619627714157104 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9435329437255859 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5126654505729675 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.783437192440033 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0Ãës„=5þz 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02296353690326214 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_5‚1 % _timestamp‚1.7843551066707873e+09 _runtime‚14765.365083231 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7760972380638123 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6168036460876465 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25477221608161926 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.5959442853927612 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2916331887245178 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2455238401889801 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2918345034122467 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30241066217422485 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.791361391544342 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807711005210876 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13852162659168243 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7177329063415527 *jet/accuracy_1_of_1‚0.3512841761112213 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7339175343513489 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0377722792327404 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7462345957756042 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.694096281948317 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023817570880055428 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03775270655751228 +jet/accuracy_2_of_2‚0.12640392780303955 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7240077257156372 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_1_of_2‚0.7902440428733826 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6611286401748657 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7127764821052551 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚42 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09789977967739105 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3026236891746521 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7677456140518188 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7144554853439331 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9968254566192627 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6306747198104858 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16995389759540558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24339736998081207 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6435505747795105 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7888989448547363 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2572992146015167 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.055872973054647446 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8759808540344238 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24092835187911987 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2921435236930847 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34390491247177124 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.566982090473175 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24516884982585907 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8323884606361389 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13999192416667938 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02035684697329998 jet/accuracy_4_of_5‚0 trainer/global_step‚17199 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36677446961402893 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02313581295311451 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6592488288879395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 #particle/accuracy‚ 0.83349609375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6849276423454285 jet/accuracy_3_of_5‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.731759786605835 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7926696538925171 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24321620166301727 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6329706311225891 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8759808540344238 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6837536692619324 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.302450567483902 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5120308995246887 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8286658525466919 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2921435236930847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.050458114594221115 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 +jet/accuracy_4_of_4‚0.01670061983168125 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3376815915107727 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.383461058139801 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17612631618976593 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7943505644798279 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15501584112644196 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.653121829032898 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10812331736087799 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.695926308631897 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6739528775215149 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22003750503063202 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.023589905351400375 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3802171051502228 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21916832029819489 +particle/specificity‚0.8984630107879639 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6428263783454895 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9806751012802124 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.694586455821991 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12087900191545486 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7688578367233276 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5752907395362854 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7743523716926575 _step‚1418 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9289266467094421 +jet/accuracy_3_of_4‚0.04078979045152664 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7619627714157104 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9435329437255859 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5126654505729675 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.783437192440033 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28524699807167053 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09744232147932053 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23933406174182892 particle/accuracy_3_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32151034474372864 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28375694155693054 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7161023020744324 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7320885062217712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7774187922477722 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.039887648075819016 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15315505862236023 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6728954911231995 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6381608247756958 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5691252946853638 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6428263783454895 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22736813127994537 particle/accuracy_2_of_4‚1 /particle/accuracy_4_of_4‚0.9426373243331909 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8649281859397888 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.535254180431366 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25622299313545227 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_3_of_3‚0.03835988789796829 'particle/f_score‚0.6058635711669922 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02035684697329998 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6576772928237915 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5512044429779053 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7188358902931213 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_5_of_5‚0Š ÀBB$ž5‚(:“ ¬²ìÒÖý£• gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.404199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òx1-‘5‚(:¢ ¬²ìÒÍ…ß—" disk.nvme0c0n1.in‚ 24598.546875& disk.nvme0c0n1.out‚33972.435546875 disk.dm-0.in‚24584.51171875 disk.sda.in‚539.4267578125 disk.sda3.out‚ 1508.30859375 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 231.8203125 disk.dm-1.out‚ 1508.30859375" proc.memory.rssMB‚ 2796.6640625) disk./.usagePercent‚39.30031589486471 disk.sda3.in‚ 415.486328125 disk.sda1.out‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 378.9296875$ disk.nvme0n1.out‚33972.435546875$ disk./.usageGB‚585.6031074523926 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 46.9111328125 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625! disk.dm-0.out‚33972.435546875 disk.sda.out‚ 1678.84375 disk.nvme0n1.in‚ 24598.546875 disk.rbd1.in‚ 2.1796875* proc.memory.percent‚17.069482803344727 network.recv‚ 3.945471e+06 proc.cpu.threads‚17 network.sent‚ 2.7597618e+07 disk.sda2.in‚ 82.544921875$ memory_percent‚97.68719673156738 cpu‚26.191677751843304­è‚²‘’5‚(:† »²ìÒÃèö” gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úùñÓ0“5‚(:¥ »²ìÒ窖 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 415.486328125 disk.dm-1.out‚ 1509.01171875$ disk./.usageGB‚585.6027221679688 disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0 network.sent‚ 2.7654617e+07 disk.sda.in‚539.4267578125 disk.rbd1.out‚ 46.9111328125& proc.memory.availableMB‚ 379.421875 cpu‚26.191202147459965 disk.sda2.in‚ 82.544921875% disk.nvme0n1.out‚33978.9697265625 disk.rbd0.in‚ 0.16015625# disk.nvme0c0n1.in‚ 24737.9140625* proc.memory.percent‚17.069482803344727 proc.cpu.threads‚17) disk./.usagePercent‚39.30029003810375 disk.sda3.out‚ 1509.01171875 disk.rbd0.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚33978.9697265625" proc.memory.rssMB‚ 2796.6640625 network.recv‚ 3.94999e+06" disk.dm-0.out‚33978.9697265625 disk.sda.out‚ 1679.546875# memory_percent‚97.6841926574707! disk.nvme0n1.in‚ 24737.9140625 disk.sda1.in‚ 8.759765625 disk.dm-0.in‚24723.91015625 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625†õGÓ4”5/ _wandbruntime‚14804  _runtime‚14804¼©6Ì•5Æ ' lr-AdamW/pg2‚2.1477383258629156e-05 trainer/global_step‚17249 % _timestamp‚1.7843551461691008e+09 _runtime‚14804.862633823 _step‚1419 ' lr-AdamW/pg1‚2.1477383258629156e-05\,ÊõÑ–5Ë % _timestamp‚1.7843551461691008e+09 _runtime‚14804.862633823 _step‚1419 ' lr-AdamW/pg1‚2.1477383258629156e-05 ' lr-AdamW/pg2‚2.1477383258629156e-05 trainer/global_step‚17249‹ rçuV4—5/ _wandbruntime‚14804  _runtime‚14804"Û…ëD˜5P TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159816618 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18253066978816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136844498 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚17249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18449378385408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18253066978816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2609416891813888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1420 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2622914968100864 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2627642101583297 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2627642459172547 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2641168018040320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159806641 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595903552877056 % _timestamp‚1.7843551461701279e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159816618 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18253408317952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577454283778048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67298891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159806641 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136844500 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2622914951061504 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18253408317952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577454265165824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2641168018040320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159806641 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595903552877056 % _timestamp‚1.7843551461701279e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159816618 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18253408317952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577454283778048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67298891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159806641 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136844500 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2622914951061504 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18253408317952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577454265165824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18253066978816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Œ ÂÌ“š5‚(:ˆ ʲìÒÔÄ“ gpu.0.gpu‚42.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýigÛ7›5‚(:¬ ʲìÒý৖) disk./.usagePercent‚39.30031666288731 disk.drbd2165.out‚0# disk.dm-0.out‚33981.98583984375 disk.sda2.in‚ 82.544921875( proc.memory.availableMB‚ 379.18359375* proc.memory.percent‚17.069482803344727 network.sent‚ 2.7659771e+07 network.recv‚ 3.953304e+06 disk.sda1.out‚0 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚24817.10546875 disk.dm-0.in‚ 24803.0703125 disk.sda.in‚539.4267578125 disk.sda3.out‚ 1509.49609375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚33981.98583984375 disk.sda.out‚ 1680.03125 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚33981.98583984375" proc.memory.rssMB‚ 2796.6640625 cpu‚26.190722307063982$ memory_percent‚97.68564701080322 disk.dm-1.out‚ 1509.49609375 proc.cpu.threads‚17 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 46.9736328125 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625" disk.nvme0n1.in‚24817.10546875$ disk./.usageGB‚585.6031188964844 disk.sda3.in‚ 415.486328125SÅØÜ4œ5/ _wandbruntime‚14805  _runtime‚14805™ê”'5‹N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30217233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595938007372800 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2641316630082560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2623062192091136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18254079605760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159815693 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2623062209130496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2609563385446400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159825684 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18226578412493 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2623062192091136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296667823 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577487693924352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2641316630082560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296677816 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2623062209130496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159825684 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595938112457728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2627789980898253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18226237862491 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136852130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2609563402485760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843551467846315e+09 _step‚1421 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37085199 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18450399921152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67302432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37085454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577487712536576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67302690 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚14805.478374912Egñž5N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136852130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2609563402485760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843551467846315e+09 _step‚1421 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37085199 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18450399921152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67302432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37085454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577487712536576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67302690 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚14805.478374912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30217233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595938007372800 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2641316630082560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2623062192091136 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18254079605760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159815693 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2623062209130496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2609563385446400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159825684 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18226578412493 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2623062192091136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296667823 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577487693924352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2641316630082560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296677816 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2623062209130496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159825684 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595938112457728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2627789980898253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18226237862491 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 §×êY4Ÿ5/ _wandbruntime‚14805  _runtime‚14805agÍ{ 5‹ 0loss/FRt2/assignment_loss‚0.7476142400940413 /loss/FRt1/detection_loss‚0.5510935074391861 /loss/FRt2/detection_loss‚1.2049955176821623 1loss/SRqqt1/detection_loss‚0.8502263871722413 _runtime‚14805.480706943 _step‚1422 epoch‚43 2loss/SRqqt1/assignment_loss‚0.4894585628584715 %loss/total_loss‚7.087245729516634 % _timestamp‚1.7843551467874224e+09 1loss/SRqqt2/detection_loss‚0.5721137042252452 trainer/global_step‚17249 0loss/FRt1/assignment_loss‚1.5378161977672782 3loss/SRqqt2/assignment_loss‚0.40498244569643277 4loss/FBt1/assignment_loss‚0.00036120675977890926 3loss/FBt2/assignment_loss‚8.400204252488075e-05 /loss/FBt1/detection_loss‚0.3319299138448615 /loss/FBt2/detection_loss‚0.3965700439344099Èà@¡5 3loss/SRqqt2/assignment_loss‚0.40498244569643277 4loss/FBt1/assignment_loss‚0.00036120675977890926 3loss/FBt2/assignment_loss‚8.400204252488075e-05 /loss/FBt1/detection_loss‚0.3319299138448615 /loss/FBt2/detection_loss‚0.3965700439344099 0loss/FRt2/assignment_loss‚0.7476142400940413 /loss/FRt1/detection_loss‚0.5510935074391861 /loss/FRt2/detection_loss‚1.2049955176821623 1loss/SRqqt1/detection_loss‚0.8502263871722413 _runtime‚14805.480706943 _step‚1422 epoch‚43 2loss/SRqqt1/assignment_loss‚0.4894585628584715 %loss/total_loss‚7.087245729516634 % _timestamp‚1.7843551467874224e+09 1loss/SRqqt2/detection_loss‚0.5721137042252452 trainer/global_step‚17249 0loss/FRt1/assignment_loss‚1.5378161977672782Ž ñžb瞢5‚(:“ Ù²ìÒ¡ƒÏ“ gpu.0.gpu‚57.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é/Ë.£5‚(:£ Ù²ìÒž§õ•' disk.nvme0c0n1.out‚33985.8544921875 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 231.8203125 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875" proc.memory.rssMB‚ 2796.6640625 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625" disk.nvme0n1.in‚24817.10546875 disk.sda3.out‚ 1509.96875 network.recv‚ 3.95777e+06 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1509.96875 disk.sda2.out‚ 170.53515625 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚24817.10546875 cpu‚26.190244873777715 disk.drbd2165.out‚0" disk.dm-0.out‚33985.8544921875 disk.sda3.in‚ 415.486328125 disk.sda1.out‚0 disk.sda.out‚ 1680.50390625* proc.memory.percent‚17.069482803344727% disk.nvme0n1.out‚33985.8544921875 disk.rbd1.out‚ 46.9736328125 network.sent‚ 2.7712076e+07$ disk./.usageGB‚585.6031761169434 disk.dm-0.in‚ 24803.0703125 disk.sda.in‚539.4267578125) disk./.usagePercent‚39.30032050300032# memory_percent‚97.6874828338623' proc.memory.availableMB‚ 378.8828125ˆ“¤5‚(:ˆ è²ìÒ¹­“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðÿ†W1¥5‚(:¦ è²ìÒ•¦°– disk.drbd2165.out‚0 disk.sda.in‚539.4267578125 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875 network.recv‚ 3.961448e+06 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚24817.10546875" disk.nvme0n1.in‚24817.10546875 disk.sda3.out‚ 1510.296875" disk.dm-0.out‚33988.9169921875 disk.sda.out‚ 1680.83203125 disk.rbd1.out‚ 47.0986328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚ 231.8203125% disk.nvme0n1.out‚33988.9169921875$ memory_percent‚97.68757820129395$ disk./.usageGB‚585.6004829406738 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625 network.sent‚ 2.7764492e+07' proc.memory.availableMB‚ 378.8671875" proc.memory.rssMB‚ 2796.6640625) disk./.usagePercent‚39.30013976168115 disk.dm-1.out‚ 1510.296875' disk.nvme0c0n1.out‚33988.9169921875 cpu‚26.18974846919857 disk.rbd0.in‚ 0.16015625 disk.dm-0.in‚ 24803.0703125 disk.sda3.in‚ 415.486328125* proc.memory.percent‚17.069482803344727è@šš4¦5/ _wandbruntime‚14844  _runtime‚14844ÂðbƒÌ§5Æ ' lr-AdamW/pg1‚2.0715891564251768e-05 ' lr-AdamW/pg2‚2.0715891564251768e-05 trainer/global_step‚17299 % _timestamp‚1.7843551860590608e+09 _runtime‚14844.752662663 _step‚1423ì^_"Ѩ5Ë ' lr-AdamW/pg1‚2.0715891564251768e-05 ' lr-AdamW/pg2‚2.0715891564251768e-05 trainer/global_step‚17299 % _timestamp‚1.7843551860590608e+09 _runtime‚14844.752662663 _step‚1423 óf4©5/ _wandbruntime‚14844  _runtime‚14844[‰ô(ª5P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2648581123402240 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2630277019582464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2648581123402240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚597626373712896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160271070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67489792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160271070 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579125703092224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18275843295677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18500565528576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2635018251663805 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137226098 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37187054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843551860602164e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2616742425407488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160261093 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18303762480640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137226098 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2630277002543104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2648580765023744 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2648580765023744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160271070 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67489792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160271070 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579125703092224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18275843295677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18500565528576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2635018251663805 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137226098 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37187054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843551860602164e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2616742425407488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160261093 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18303762480640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137226098 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2630277002543104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160270145 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18305116453888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30304483 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚597660823740928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18501586805248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18305116453888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67493785 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2635165773389399 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2630424260612096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67493534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚14845.367674176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch|Æö±Ý_end/requested_bytes.small_pool.freed‚18276854349399 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2648729377065984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2630424260612096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1425 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18304775107584 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616888936079360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635166130978761 trainer/global_step‚17299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579159150462976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297503875 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513868 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2630424243572736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137233732 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137233730 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2616888919040000 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18304775107584ÿa='®5N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18305116453888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30304483 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚597660823740928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18501586805248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18305116453888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67493785 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2635165773389399 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2630424260612096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67493534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚14845.367674176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18276854349399 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2648729377065984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2630424260612096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1425 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18304775107584 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616888936079360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635166130978761 trainer/global_step‚17299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579159150462976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297503875 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513868 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2630424243572736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137233732 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137233730 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2616888919040000 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18304775107584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160270145‘ ®ƒ;_4¯5/ _wandbruntime‚14845  _runtime‚14845škI°5‰ 2loss/SRqqt1/assignment_loss‚0.4371387051985156 /loss/FRt2/detection_loss‚1.1742797599303538 1loss/SRqqt1/detection_loss‚0.8042205484242647 1loss/SRqqt2/detection_loss‚0.5320132898193196 _step‚1426 0loss/FRt2/assignment_loss‚0.8205994648020525 /loss/FRt1/detection_loss‚0.5203824000146916 /loss/FBt1/detection_loss‚0.3112355554576425 epoch‚43 _runtime‚14845.371406602 0loss/FRt1/assignment_loss‚1.4867423748028255 2loss/SRqqt2/assignment_loss‚0.4412040993664458 %loss/total_loss‚6.847859795870459 4loss/FBt1/assignment_loss‚0.00020843580106921777 3loss/FBt2/assignment_loss‚0.0007256309883407752 /loss/FBt2/detection_loss‚0.3191095312649365 trainer/global_step‚17299 $ _timestamp‚1.784355186677763e+09=‚ó±5Ž %loss/total_loss‚6.847859795870459 4loss/FBt1/assignment_loss‚0.00020843580106921777 3loss/FBt2/assignment_loss‚0.0007256309883407752 /loss/FBt2/detection_loss‚0.3191095312649365 trainer/global_step‚17299 $ _timestamp‚1.784355186677763e+09 2loss/SRqqt1/assignment_loss‚0.4371387051985156 /loss/FRt2/detection_loss‚1.1742797599303538 1loss/SRqqt1/detection_loss‚0.8042205484242647 1loss/SRqqt2/detection_loss‚0.5320132898193196 _step‚1426 0loss/FRt2/assignment_loss‚0.8205994648020525 /loss/FRt1/detection_loss‚0.5203824000146916 /loss/FBt1/detection_loss‚0.3112355554576425 epoch‚43 _runtime‚14845.371406602 0loss/FRt1/assignment_loss‚1.4867423748028255 2loss/SRqqt2/assignment_loss‚0.4412040993664458’ (bD†“²5‚(:ˆ ÷²ìÒÖͪ“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚203.631' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.7262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0. QC4³5‚(:© ÷²ìÒž™˜ disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚24817.10546875% proc.memory.availableMB‚ 379.09375( disk.nvme0c0n1.out‚33993.20947265625 disk.sda.out‚ 1681.15625 disk.sda3.out‚ 1510.62109375 cpu‚26.189281946253683 network.sent‚ 2.7770077e+07) disk./.usagePercent‚39.30013771362088 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.sda2.out‚ 170.53515625" proc.memory.rssMB‚ 2796.6640625$ disk./.usageGB‚585.6004524230957 disk.dm-1.in‚ 231.8203125 disk.dm-0.in‚ 24803.0703125 disk.sda.in‚539.4267578125 disk.sda3.in‚ 415.486328125 network.recv‚ 3.964998e+06 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1510.62109375" disk.nvme0n1.in‚24817.10546875 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0# disk.dm-0.out‚33993.20947265625 disk.rbd1.out‚ 47.0986328125 disk.drbd2165.in‚ 2.25390625 disk.sda2.in‚ 82.544921875& disk.nvme0n1.out‚33993.20947265625$ memory_percent‚97.68619537353516* proc.memory.percent‚17.069482803344727 proc.cpu.threads‚17I+4·œ´5‚(:‘ †³ìÒíØ„” gpu.0.gpu‚28.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ºe/µ5‚(:¤ †³ìÒÈäõ– disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.dm-1.out‚1510.875# disk.dm-0.out‚33996.88720703125 disk.sda.out‚ 1681.41015625$ memory_percent‚97.68855571746826 proc.cpu.threads‚17 network.recv‚ 3.969158e+06$ disk./.usageGB‚585.5909156799316 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.8203125( disk.nvme0c0n1.out‚33996.88720703125 disk.dm-0.in‚ 24945.2421875& disk.nvme0n1.out‚33996.88720703125 disk.sda3.out‚1510.875 disk.sda.in‚539.4267578125 disk.rbd1.out‚ 47.2236328125 disk.sda3.in‚ 415.486328125 cpu‚26.18882712634793) disk./.usagePercent‚39.29949769478528 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚24959.27734375* proc.memory.percent‚17.069482803344727 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625( proc.memory.availableMB‚ 378.70703125" proc.memory.rssMB‚ 2796.6640625 network.sent‚ 2.7832602e+07$ disk.nvme0c0n1.in‚24959.27734375›(B¶5‚(:’ •³ìÒÃäË“ gpu.0.gpu‚46.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žœá_2·5‚(:§ •³ìÒÇϘ• disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.dm-0.in‚ 24945.2421875 proc.cpu.threads‚17 network.sent‚ 2.7837794e+07 disk.dm-1.out‚ 1511.046875" disk.dm-0.out‚34001.3388671875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚24959.27734375' disk.nvme0c0n1.out‚34001.3388671875 disk.sda2.in‚ 82.544921875 network.recv‚ 3.972784e+06 disk.sda.out‚ 1681.58203125 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚24959.27734375% disk.nvme0n1.out‚34001.3388671875* proc.memory.percent‚17.069482803344727 disk.drbd2165.in‚ 2.25390625 disk.rbd1.out‚ 47.2236328125$ memory_percent‚97.68707752227783) disk./.usagePercent‚39.29949180661199$ disk./.usageGB‚585.5908279418945 disk.sda.in‚539.4267578125 disk.sda3.in‚ 415.486328125 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 378.94921875" proc.memory.rssMB‚ 2796.6640625 cpu‚26.18835840072014 disk.dm-1.in‚ 231.8203125 disk.sda3.out‚ 1511.046875knð4¸5/ _wandbruntime‚14884  _runtime‚14884Dô[̹5Æ ' lr-AdamW/pg1‚1.9967491013299176e-05 ' lr-AdamW/pg2‚1.9967491013299176e-05 trainer/global_step‚17349 % _timestamp‚1.7843552261161523e+09 _runtime‚14884.809661748 _step‚1427¯ˆ5,Ѻ5Ë % _timestamp‚1.7843552261161523e+09 _runtime‚14884.809661748 _step‚1427 ' lr-AdamW/pg1‚1.9967491013299176e-05 ' lr-AdamW/pg2‚1.9967491013299176e-05 trainer/global_step‚17349“ Ÿ \u4»5/ _wandbruntime‚14884  _runtime‚14884Õ(wà$¼5P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2655993512007168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2655993512007168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67680355 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298323243 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18354457982464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37290367 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137607698 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2637639054024704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18354799321600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2637639054024704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160715545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2637639071064064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18326459782585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18354799321600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137607698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580797141018624 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2642394759333563 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2637639071064064 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚17349 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 !PÝB'XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30389991 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160715545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2624067959001088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269×´+ (½5†P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2637639071064064 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚17349 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30389991 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160715545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2624067959001088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2655993512007168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2655993512007168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67680355 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298323243 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18354457982464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37290367 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137607698 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2637639054024704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18354799321600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2637639054024704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160715545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2637639071064064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18326459782585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18354799321600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137607698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580797141018624 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2642394759333563 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976” Ê‹¦É4¾5/ _wandbruntime‚14885  _runtime‚14885¤Šî>'¿5ŠN LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1429 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2637786295054336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843552267300656e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2637786295054336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67684430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642542281059269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137615332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67684185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2624214469672960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160734588 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580830569777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160734588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160724597 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18327811386309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚73889792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580830588389376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599383484604416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298349920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2656141765663744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚17349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18355811955712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160724597 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18552914827264 Bæ'À5N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30391736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18552988717056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599383577106432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137615330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30391733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2642541923469907 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18355470609408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137615330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37292452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2656142124049408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37292694 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327470836307 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2637786312093696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18355811955712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18355470609408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580830588389376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599383484604416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298349920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2656141765663744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚17349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18355811955712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160724597 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18552914827264 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2637786295054336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843552267300656e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2637786295054336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67684430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642542281059269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137615332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67684185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2624214469672960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160734588 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580830569777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160734588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160724597 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18327811386309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚73889792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FÎ_5Å5‚(:ª ¤³ìÒõ÷•$ disk./.usageGB‚585.5920944213867 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625# disk.dm-0.out‚34005.55029296875 disk.sda.out‚ 1681.9375" proc.memory.rssMB‚ 2796.6640625 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 231.8203125 disk.sda.in‚539.4267578125 disk.rbd1.out‚ 47.2861328125$ memory_percent‚97.68946170806885 proc.cpu.threads‚17) disk./.usagePercent‚39.29957680111335& disk.nvme0n1.out‚34005.55029296875 disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚24959.27734375( disk.nvme0c0n1.out‚34005.55029296875 disk.dm-0.in‚ 24945.2421875 disk.dm-1.out‚ 1511.40234375 disk.sda2.out‚ 170.53515625( proc.memory.availableMB‚ 378.55859375 network.sent‚ 2.7843386e+07 disk.rbd0.out‚0 disk.sda3.out‚ 1511.40234375 network.recv‚ 3.975885e+06 disk.sda1.out‚0 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚24959.27734375 disk.sda3.in‚ 415.486328125* proc.memory.percent‚17.069482803344727 cpu‚26.18786480441973Š˜éÀ“Æ5‚(:ˆ ³³ìÒ†ñ·“ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÞN.Ç5‚(:£ ³³ìÒÔ…Ü•$ disk./.usageGB‚585.5919151306152" disk.nvme0n1.in‚24959.27734375$ memory_percent‚97.68819808959961 cpu‚26.187392495772876( disk.nvme0c0n1.out‚34012.19873046875 disk.sda2.in‚ 82.544921875 disk.sda3.out‚1511.875) disk./.usagePercent‚39.29956476875925 disk.sda1.out‚0 disk.sda.in‚539.4267578125 disk.sda2.out‚ 170.53515625& proc.memory.availableMB‚ 378.765625 disk.drbd2165.out‚0# disk.dm-0.out‚34012.19873046875 disk.sda.out‚ 1682.41015625 disk.rbd1.out‚ 47.2861328125 network.recv‚ 3.980617e+06 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚24959.27734375 disk.rbd1.in‚ 2.1796875" proc.memory.rssMB‚ 2796.6640625* proc.memory.percent‚17.069482803344727 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚1511.875& disk.nvme0n1.out‚34012.19873046875 disk.sda3.in‚ 415.486328125 disk.rbd0.in‚ 0.16015625 disk.dm-0.in‚ 24945.2421875 proc.cpu.threads‚17 network.sent‚ 2.7906554e+07 disk.dm-1.in‚ 231.8203125c™ïn4È5/ _wandbruntime‚14924  _runtime‚14924„DЊÌÉ5Æ ' lr-AdamW/pg1‚1.9232229674140148e-05 ' lr-AdamW/pg2‚1.9232229674140148e-05 trainer/global_step‚17399 % _timestamp‚1.7843552659702303e+09 _runtime‚14924.663798309 _step‚1431!¶~ÑÊ5Ë ' lr-AdamW/pg1‚1.9232229674140148e-05 ' lr-AdamW/pg2‚1.9232229674140148e-05 trainer/global_step‚17399 % _timestamp‚1.7843552659702303e+09 _runtime‚14924.663798309 _step‚1431— —‹Ýë4Ë5/ _wandbruntime‚14924  _runtime‚14924O‚ä…(Ì5P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299169274 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137989298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2663406617369088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2649770551824821 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2631393492594688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚17399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37393765 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18405494823424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161179974 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18603255557120 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2645001105506304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚14924.665016158 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18405153484288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18405153484288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18603339939840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚1432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30477238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18405494823424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2663406258990592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161179974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601071834502144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161169997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2631393475555328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37394017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582468578945024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18377076269493 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2645001122545664 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2645001105506304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582468597557248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161169997 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30477241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2663406258990592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161179974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601071834502144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161169997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2631393475555328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37394017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582468578945024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18377076269493 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2645001122545664 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2645001105506304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582468597557248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161169997 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30477241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299169274 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137989298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2663406617369088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2649770551824821 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2631393492594688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚17399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37393765 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18405494823424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161179974˜ £B¯^žÎ5‚(:“ ³ìÒ©É´“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûç®õ6Ï5‚(:« ³ìÒ€óŒ–" disk.nvme0c0n1.in‚ 25122.578125 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0 disk.sda3.out‚ 1512.20703125* proc.memory.percent‚17.069482803344727$ disk./.usageGB‚585.5929794311523 disk.drbd2165.in‚ 2.25390625 disk.nvme0n1.in‚ 25122.578125 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.3486328125 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1512.20703125) disk./.usagePercent‚39.29963619486131# disk.dm-0.out‚34017.33349609375 disk.sda1.out‚0" proc.memory.rssMB‚ 2796.6640625 proc.cpu.threads‚17 network.sent‚ 2.7911116e+07 disk.dm-0.in‚25108.54296875 disk.dm-1.in‚ 231.8203125$ memory_percent‚97.69117832183838( proc.memory.availableMB‚ 378.27734375 cpu‚26.186918894011633( disk.nvme0c0n1.out‚34017.33349609375 disk.sda.in‚539.4267578125& disk.nvme0n1.out‚34017.33349609375 network.recv‚ 3.983995e+06 disk.sda2.in‚ 82.544921875 disk.drbd2165.out‚0 disk.sda.out‚ 1682.7421875 disk.sda3.in‚ 415.486328125M¨„t4Ð5/ _wandbruntime‚14925  _runtime‚14925míËÑ5ŠN VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18406166111232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2663554871032832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2645148346535936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601106399509504 % _timestamp‚1.7843552665857453e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161179049 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18406507457536 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175979 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299185972 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2645148346535936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18604373193728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2649918073550415 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30478983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601106309104640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2649918431139777 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18378427873217 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137996932 trainer/global_step‚17399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161189040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18604301401088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚14925.279494728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1433 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137996932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceѾü³E StatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2663554871032832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137996930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2631539986227200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175979 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67875013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30478986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582502026315776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37395767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2663554512647168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37396027 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161189040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18406507457536 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18378087323215 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67874750 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2645148363575296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888a9ëŒ'Ò5N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299185972 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2645148346535936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18604373193728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2649918073550415 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30478983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601106309104640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2649918431139777 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18378427873217 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137996932 trainer/global_step‚17399 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161189040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18604301401088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚14925.279494728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185972 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1433 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137996932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2663554871032832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137996930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2631539986227200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175979 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67875013 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30478986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582502026315776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37395767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2663554512647168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37396027 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161189040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18406507457536 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚90404864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18378087323215 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67874750 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2645148363575296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18406166111232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2663554871032832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2645148346535936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601106399509504 % _timestamp‚1.7843552665857453e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161179049 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18406507457536 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175979™ ¹º¶ñ4Ó5/ _wandbruntime‚14925  _runtime‚14925lN¹Ô5ˆ /loss/FRt1/assignment_loss‚1.548173736368689 /loss/FRt1/detection_loss‚0.5370888782241026 2loss/FBt2/assignment_loss‚0.003086925390027554 /loss/FBt2/detection_loss‚0.3702118631551305 %loss/total_loss‚6.997906841307016 epoch‚43 trainer/global_step‚17399 _step‚1434 0loss/FRt2/assignment_loss‚0.6405811067465897 3loss/SRqqt2/assignment_loss‚0.39664068062769076 3loss/FBt1/assignment_loss‚0.0001020925453167898 /loss/FRt2/detection_loss‚1.1444495910994628 1loss/SRqqt1/detection_loss‚0.8766974036446467 1loss/SRqqt2/detection_loss‚0.5545933205454151 % _timestamp‚1.7843552665887914e+09 2loss/SRqqt1/assignment_loss‚0.5020661172783828 /loss/FBt1/detection_loss‚0.4242151256815608 _runtime‚14925.282075099‘åüÕ5 2loss/FBt2/assignment_loss‚0.003086925390027554 /loss/FBt2/detection_loss‚0.3702118631551305 %loss/total_loss‚6.997906841307016 epoch‚43 trainer/global_step‚17399 _step‚1434 0loss/FRt2/assignment_loss‚0.6405811067465897 3loss/SRqqt2/assignment_loss‚0.39664068062769076 3loss/FBt1/assignment_loss‚0.0001020925453167898 /loss/FRt2/detection_loss‚1.1444495910994628 1loss/SRqqt1/detection_loss‚0.8766974036446467 1loss/SRqqt2/detection_loss‚0.5545933205454151 % _timestamp‚1.7843552665887914e+09 2loss/SRqqt1/assignment_loss‚0.5020661172783828 /loss/FBt1/detection_loss‚0.4242151256815608 _runtime‚14925.282075099 /loss/FRt1/assignment_loss‚1.548173736368689 /loss/FRt1/detection_loss‚0.5370888782241026š ¬q“Ö5‚(:ˆ ѳìÒ£ñ¾“ gpu.0.gpu‚37.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¨b.×5‚(:£ ѳìÒÁ­Ä– cpu‚26.186412697143982) disk./.usagePercent‚39.30016613045718 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875 proc.cpu.threads‚17 disk.sda3.in‚ 415.486328125 disk.sda1.out‚0* proc.memory.percent‚17.069482803344727" disk.nvme0c0n1.in‚ 25122.578125' disk.nvme0c0n1.out‚34025.3916015625$ memory_percent‚97.68848419189453$ disk./.usageGB‚585.6008758544922 disk.sda.in‚539.4267578125 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 47.3486328125" proc.memory.rssMB‚ 2796.6640625 network.sent‚ 2.796416e+07 network.recv‚ 3.988491e+06 disk.dm-0.in‚25108.54296875 disk.rbd0.out‚0 disk.sda.out‚ 1683.15234375 disk.dm-1.in‚ 231.8203125 disk.dm-1.out‚ 1512.6171875% proc.memory.availableMB‚ 378.71875" disk.dm-0.out‚34025.3916015625 disk.nvme0n1.in‚ 25122.578125% disk.nvme0n1.out‚34025.3916015625 disk.sda3.out‚ 1512.6171875 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625%X ;’Ø5‚(:‡ à³ìÒ¹¹¢“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!·÷ )Ù5‚(:ž à³ìÒåõ•* proc.memory.percent‚17.069482803344727 proc.cpu.threads‚17 network.sent‚ 2.801659e+07 network.recv‚ 3.992334e+06 disk.rbd0.out‚0% disk.nvme0c0n1.out‚34028.51953125 disk.sda.in‚539.4267578125$ memory_percent‚97.68962860107422 disk.sda1.in‚ 8.759765625" disk.nvme0c0n1.in‚ 25122.578125$ disk./.usageGB‚585.6008491516113 disk.sda1.out‚0# disk.nvme0n1.out‚34028.51953125 disk.sda3.out‚ 1513.17578125) disk./.usagePercent‚39.30016433840444 disk.dm-1.out‚ 1513.17578125 disk.dm-0.out‚34028.51953125 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 25122.578125 disk.sda3.in‚ 415.486328125 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.sda.out‚ 1683.7109375 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 47.4111328125 cpu‚26.185961368465417 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚25108.54296875% proc.memory.availableMB‚ 378.53125" proc.memory.rssMB‚ 2796.6640625û/Ú 4Ú5/ _wandbruntime‚14964  _runtime‚14964/ÕåÌÛ5Æ ' lr-AdamW/pg1‚1.8510154771236606e-05 ' lr-AdamW/pg2‚1.8510154771236606e-05 trainer/global_step‚17449 % _timestamp‚1.7843553060288346e+09 _runtime‚14964.722414167 _step‚1435P—³ÛÑÜ5Ë ' lr-AdamW/pg1‚1.8510154771236606e-05 ' lr-AdamW/pg2‚1.8510154771236606e-05 trainer/global_step‚17449 % _timestamp‚1.7843553060288346e+09 _runtime‚14964.722414167 _step‚1435› Áÿñ4Ý5/ _wandbruntime‚14964  _runtime‚14964W…³:(Þ5P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2652363156987904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2670819005974016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18455848986112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18654754165248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18455848986112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37497946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2657147059494579 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161624449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18456190325248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843553060299137e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37497685 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2652363174027264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30564491 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68062437 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138370898 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161634426 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2670819005974016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138370900 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2652363174027264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138370900 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18428033306291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161634426 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602794688751104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚17449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18654671879680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18427692756401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2638719009148928 _runtime‚14964.723642045 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138370898 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2670819005974016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18455848986112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18654754165248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18455848986112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37497946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2657147059494579 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161624449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18456190325248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843553060299137e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37497685 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2652363174027264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30564491 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68062437 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138370898 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161634426 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2670819005974016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138370900 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2652363174027264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138370900 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18428033306291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161634426 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602794688751104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚17449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18654671879680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18427692756401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2638719009148928 _runtime‚14964.723642045 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138370898 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68066045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37500060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2670967618016256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18456861613056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300012031 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161633501 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138378532 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18456861613056 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138378530 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2670967618016256 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2652510398017536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18655697094656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022024 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300012031 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18428703810123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2652510398017536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18457202959360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30566236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18655779373056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138378530 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138378532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18457202959360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2638865536860160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2652510415056896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18429044360125 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7843553066436732e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2670967259630592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚584173445629952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983Ž©©u'â5N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7843553066436732e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2670967259630592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚584173445629952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68066045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37500060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2670967618016256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18456861613056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300012031 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161633501 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138378532 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18456861613056 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138378530 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2670967618016256 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2652510398017536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18655697094656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022024 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300012031 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18428703810123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2652510398017536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18457202959360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30566236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18655779373056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138378530 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138378532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18457202959360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2638865536860160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2652510415056896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18429044360125 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 °îg÷4ã5/ _wandbruntime‚14965  _runtime‚14965ÇÒBä5Š _step‚1438 3loss/FBt2/assignment_loss‚2.571344674044023e-05 /loss/FRt2/detection_loss‚1.1483177801981612 1loss/SRqqt2/detection_loss‚0.6658836011463495 _runtime‚14965.341216892 1loss/SRqqt1/detection_loss‚0.7614211238906361 %loss/total_loss‚6.951482735982675 trainer/global_step‚17449 0loss/FRt1/assignment_loss‚1.4948474855370861 3loss/SRqqt1/assignment_loss‚0.49270924851743075 2loss/FBt1/assignment_loss‚0.016955511643189425 epoch‚43 0loss/FRt2/assignment_loss‚0.7511521901845489 3loss/SRqqt2/assignment_loss‚0.42998004171011506 /loss/FRt1/detection_loss‚0.5380846910493475 /loss/FBt1/detection_loss‚0.2799505266611959 /loss/FBt2/detection_loss‚0.3721548219978745 % _timestamp‚1.7843553066479013e+09@¾!å5 /loss/FRt1/detection_loss‚0.5380846910493475 /loss/FBt1/detection_loss‚0.2799505266611959 /loss/FBt2/detection_loss‚0.3721548219978745 % _timestamp‚1.7843553066479013e+09 _step‚1438 3loss/FBt2/assignment_loss‚2.571344674044023e-05 /loss/FRt2/detection_loss‚1.1483177801981612 1loss/SRqqt2/detection_loss‚0.6658836011463495 _runtime‚14965.341216892 1loss/SRqqt1/detection_loss‚0.7614211238906361 %loss/total_loss‚6.951482735982675 trainer/global_step‚17449 0loss/FRt1/assignment_loss‚1.4948474855370861 3loss/SRqqt1/assignment_loss‚0.49270924851743075 2loss/FBt1/assignment_loss‚0.016955511643189425 epoch‚43 0loss/FRt2/assignment_loss‚0.7511521901845489 3loss/SRqqt2/assignment_loss‚0.42998004171011506ž ª¿Å˜“æ5‚(:ˆ ï³ìҽŲ“ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žžé%ç5‚(:š ï³ìÒïÛ— disk.dm-1.out‚ 1513.6015625* disk./.usagePercent‚39.300168946540055" disk.nvme0c0n1.in‚ 25122.578125 disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0 disk.dm-0.out‚ 34032.515625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 231.8203125 network.recv‚ 3.996055e+06 disk.dm-0.in‚25108.54296875 disk.sda.out‚ 1684.13671875 disk.sda3.in‚ 415.486328125 disk.rbd1.out‚ 47.4111328125 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625 disk.sda.in‚539.4267578125 disk.sda2.in‚ 82.544921875 disk.sda1.in‚ 8.759765625! disk.nvme0n1.out‚ 34032.515625$ memory_percent‚97.69158363342285" proc.memory.rssMB‚ 2796.6640625 network.sent‚ 2.8021145e+07$ disk./.usageGB‚585.6009178161621' proc.memory.availableMB‚ 378.2109375 disk.rbd0.in‚ 0.16015625 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 25122.578125 disk.sda3.out‚ 1513.6015625* proc.memory.percent‚17.069482803344727 cpu‚26.18548775601722# disk.nvme0c0n1.out‚ 34032.515625ª[ÐÝœè5‚(:‘ þ³ìÒÄί“ gpu.0.gpu‚10.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú}«å4é5‚(:© þ³ìÒ–$ disk./.usageGB‚585.6010475158691 disk.rbd1.out‚ 47.4736328125 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1684.484375 disk.sda2.in‚ 82.544921875$ memory_percent‚97.69258499145508* proc.memory.percent‚17.069482803344727 disk.dm-1.in‚ 231.8203125 disk.drbd2165.out‚0 disk.dm-0.in‚ 25271.6796875 disk.sda.in‚539.4267578125 network.sent‚ 2.808473e+07 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1513.94921875" disk.nvme0n1.in‚25285.71484375 network.recv‚ 4.000318e+06$ disk.nvme0c0n1.in‚25285.71484375 proc.cpu.threads‚17 disk.sda3.in‚ 415.486328125 disk.rbd0.out‚0 disk.sda3.out‚ 1513.94921875 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0# disk.dm-0.out‚34036.32080078125 disk.sda2.out‚ 170.53515625& proc.memory.availableMB‚ 378.046875" proc.memory.rssMB‚ 2796.6640625 cpu‚26.18502286998468 disk.rbd1.in‚ 2.1796875( disk.nvme0c0n1.out‚34036.32080078125& disk.nvme0n1.out‚34036.32080078125) disk./.usagePercent‚39.30017765079622£VJ“ê5‚(:ˆ ´ìҾس“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ 5S7ë5‚(:¬ ´ìÒ÷Êó• disk.rbd0.in‚ 0.16015625# disk.dm-0.out‚34042.00146484375 disk.sda2.in‚ 82.544921875 network.sent‚ 2.808977e+07) disk./.usagePercent‚39.30016638646471 disk.dm-1.out‚ 1514.16796875* proc.memory.percent‚17.069482803344727$ disk./.usageGB‚585.6008796691895 disk.sda3.in‚ 415.486328125 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0 network.recv‚ 4.003751e+06" disk.nvme0n1.in‚25285.71484375 disk.dm-0.in‚ 25271.6796875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚25285.71484375$ memory_percent‚97.69418239593506( proc.memory.availableMB‚ 377.78515625 disk.sda3.out‚ 1514.16796875 disk.sda.out‚ 1684.703125 disk.sda2.out‚ 170.53515625 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625 disk.sda.in‚539.4267578125& disk.nvme0n1.out‚34042.00146484375 disk.rbd1.out‚ 47.4736328125( disk.nvme0c0n1.out‚34042.00146484375" proc.memory.rssMB‚ 2796.6640625 cpu‚26.184572857934544fcúj4ì5/ _wandbruntime‚15004  _runtime‚1500448ˆ¡Ìí5Æ _step‚1439 ' lr-AdamW/pg1‚1.7801312682110297e-05 ' lr-AdamW/pg2‚1.7801312682110297e-05 trainer/global_step‚17499 % _timestamp‚1.7843553459144778e+09 _runtime‚15004.608078797§ÝtªÑî5Ë % _timestamp‚1.7843553459144778e+09 _runtime‚15004.608078797 _step‚1439 ' lr-AdamW/pg1‚1.7801312682110297e-05 ' lr-AdamW/pg2‚1.7801312682110297e-05 trainer/global_step‚17499Ÿ ’qÈï4ï5/ _wandbruntime‚15004  _runtime‚15004*:Ëð5P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162078901 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/(qˆ‡<'allocated_bytes.small_pool.freed‚18506544487936 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2664522851985837 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18478649793199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68253218 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚585811473410048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18706044540928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18506885827072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2678232111335936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2659725208469504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18705958061056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2678232111335936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2678231752957440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831399 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138752498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659725208469504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604517517950976 _runtime‚15004.612338439 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138752500 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30651738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2678231752957440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18506544487936 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2664522851985837 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18478649793199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68253218 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚585811473410048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18706044540928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18506885827072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2678232111335936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2659725208469504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18705958061056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2678232111335936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2678231752957440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831399 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138752498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659725208469504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚604517517950976 _runtime‚15004.612338439 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138752500 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30651738 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2678231752957440 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858076 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68257290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2664670373711431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843553465293696e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659872466538496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659872449499136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2664670731300793 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300848083 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18507898461184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138760130 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162097944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300858076 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68257026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚17499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2678380006614016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848083 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15005.223097399 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18507898461184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18507557114880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2646191053414400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18479660847033 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2659872449499136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18706979780096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138760132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37603543 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0^1|~P ô5N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68257026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚17499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2678380006614016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848083 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15005.223097399 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18507898461184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18507557114880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2646191053414400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18479660847033 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2659872449499136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18706979780096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138760132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚š †;Æ1438 _step‚1441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37603543 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2678380364999680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚585844883556352 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2646191070453760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858076 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68257290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2664670373711431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843553465293696e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659872466538496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659872449499136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2664670731300793 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300848083 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18507898461184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138760130 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162097944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300858076¡ Û¡¯º4õ5/ _wandbruntime‚15005  _runtime‚15005ú‡L¶ö5Š %loss/total_loss‚7.035477301378716 epoch‚43 0loss/FRt2/assignment_loss‚0.7775122803200334 2loss/SRqqt2/assignment_loss‚0.4301014192388202 1loss/SRqqt1/detection_loss‚0.9215788471986186 trainer/global_step‚17499 _runtime‚15005.225871496 _step‚1442 3loss/FBt2/assignment_loss‚4.855740882897203e-06 /loss/FBt1/detection_loss‚0.3818597632812776 /loss/FRt1/assignment_loss‚1.488296285367781 3loss/SRqqt1/assignment_loss‚0.45250662647132683 4loss/FBt1/assignment_loss‚1.1253300355759973e-05 /loss/FRt1/detection_loss‚0.5242557633218611 /loss/FRt2/detection_loss‚1.1023698832767577 1loss/SRqqt2/detection_loss‚0.6121747206698928 /loss/FBt2/detection_loss‚0.3448056031911084 % _timestamp‚1.7843553465325587e+09î˹÷5 2loss/SRqqt2/assignment_loss‚0.4301014192388202 1loss/SRqqt1/detection_loss‚0.9215788471986186 trainer/global_step‚17499 _runtime‚15005.225871496 _step‚1442 3loss/FBt2/assignment_loss‚4.855740882897203e-06 /loss/FBt1/detection_loss‚0.3818597632812776 /loss/FRt1/assignment_loss‚1.488296285367781 3loss/SRqqt1/assignment_loss‚0.45250662647132683 4loss/FBt1/assignment_loss‚1.1253300355759973e-05 /loss/FRt1/detection_loss‚0.5242557633218611 /loss/FRt2/detection_loss‚1.1023698832767577 1loss/SRqqt2/detection_loss‚0.6121747206698928 /loss/FBt2/detection_loss‚0.3448056031911084 % _timestamp‚1.7843553465325587e+09 %loss/total_loss‚7.035477301378716 epoch‚43 0loss/FRt2/assignment_loss‚0.7775122803200334¢ ŸïÔ‡“ø5‚(:ˆ œ´ìқ⡓ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚250.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IEÃ×6ù5‚(:« œ´ìÒþÍå• network.sent‚ 2.8094963e+07 disk.rbd1.in‚ 2.1796875# memory_percent‚97.6942777633667( proc.memory.availableMB‚ 377.76953125 disk.sda3.in‚ 415.486328125 network.recv‚ 4.007069e+06 disk.drbd2165.out‚0 disk.sda.out‚ 1685.02734375 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.8203125 disk.drbd2165.in‚ 2.25390625& disk.nvme0n1.out‚34046.43896484375 disk.rbd0.in‚ 0.16015625$ disk./.usageGB‚585.6009559631348 proc.cpu.threads‚17( disk.nvme0c0n1.out‚34046.43896484375 disk.sda2.out‚ 170.53515625* proc.memory.percent‚17.069482803344727# disk.dm-0.out‚34046.43896484375 disk.dm-1.out‚ 1514.4921875 disk.dm-0.in‚ 25271.6796875 disk.sda.in‚539.4267578125" disk.nvme0n1.in‚25285.71484375 cpu‚26.184073490673352 disk.sda3.out‚ 1514.4921875 disk.rbd0.out‚0( disk./.usagePercent‚39.3001715066154 disk.sda1.out‚0$ disk.nvme0c0n1.in‚25285.71484375 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 47.5361328125" proc.memory.rssMB‚ 2796.6640625‹€ÿ“ú5‚(:ˆ «´ìÒŽ‹Ü“ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nôa6û5‚(:« «´ìÒ眖 disk.dm-1.out‚ 1514.8984375 network.recv‚ 4.011622e+06 disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚34053.73486328125& disk.nvme0n1.out‚34053.73486328125 proc.cpu.threads‚17( disk./.usagePercent‚39.3001812349017 disk.sda3.in‚ 415.486328125 disk.rbd1.out‚ 47.5361328125 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚25285.71484375 disk.sda2.in‚ 82.544921875" proc.memory.rssMB‚ 2796.6640625 network.sent‚ 2.8157501e+07 disk.sda1.out‚0$ memory_percent‚97.69425392150879 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 231.8203125 disk.sda.out‚ 1685.43359375 cpu‚26.183607095920575# disk.dm-0.out‚34053.73486328125 disk.sda.in‚539.4267578125$ disk./.usageGB‚585.6011009216309 disk.sda3.out‚ 1514.8984375 disk.dm-0.in‚ 25271.6796875 disk.rbd0.in‚ 0.16015625 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚25285.71484375' proc.memory.availableMB‚ 377.7734375* proc.memory.percent‚17.069482803344727™4ÃG4ü5/ _wandbruntime‚15044  _runtime‚15044`Y!Ëý5Å % _timestamp‚1.7843553859568965e+09 _runtime‚15044.65045471 _step‚1443 ' lr-AdamW/pg1‚1.7105748934363975e-05 ' lr-AdamW/pg2‚1.7105748934363975e-05 trainer/global_step‚17549|õ˜Ðþ5Ê _step‚1443 ' lr-AdamW/pg1‚1.7105748934363975e-05 ' lr-AdamW/pg2‚1.7105748934363975e-05 trainer/global_step‚17549 % _timestamp‚1.7843553859568965e+09 _runtime‚15044.65045471£ m&ñÂ4ÿ5/ _wandbruntime‚15044  _runtime‚15044Ff?i(€6P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚15044.651836569 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162533353 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18557239989760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18757237965824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30738991 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18557581328896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 301677430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139134098 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2685644499940864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2667087276990464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2667087259951104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68444179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587482911336448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18557581328896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2685644858319360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68444450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚606240242073600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2685644499940864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843553859581397e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2667087276990464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2653370093375488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2685644858319360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162543330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2671899002066345 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2653370076336128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18528925730217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162533353 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162543330 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18757330737152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 trainer/global_step‚17549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37705459 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18529266280107 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚1444 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139134100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚606240130690048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301667451 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139134100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139134098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2671899359655595 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976RR:ã%6†P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2685644499940864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843553859581397e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2667087276990464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2653370093375488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2685644858319360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162543330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2671899002066345 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2653370076336128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18528925730217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162533353 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162543330 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18757330737152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 trainer/global_step‚17549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37705459 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18529266280107 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚1444 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139134100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚606240130690048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301667451 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139134100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139134098 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2671899359655595 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚15044.651836569 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162533353 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18557239989760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18757237965824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30738991 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18557581328896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 301677430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139134098 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2685644499940864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2667087276990464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2667087259951104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68444179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587482911336448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18557581328896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/res"Âs)erved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2685644858319360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68444450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚606240242073600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269¤ âÝ/­ž‚6‚(:“ º´ìÒýÔå“ gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.975199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bu„Y0ƒ6‚(:¥ º´ìÒÞŒÿ– disk.sda2.out‚ 170.53515625$ disk.nvme0c0n1.in‚25440.58203125" disk.nvme0n1.in‚25440.58203125 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.recv‚ 4.015121e+06 disk.drbd2165.in‚ 2.25390625 disk.dm-1.out‚ 1515.56640625 network.sent‚ 2.8163185e+07$ disk./.usageGB‚585.6009674072266 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625! disk.dm-0.out‚34058.263671875 disk.sda.out‚ 1686.1015625 disk.dm-1.in‚ 233.44140625* proc.memory.percent‚17.069482803344727& disk.nvme0c0n1.out‚34058.263671875 disk.sda.in‚541.0478515625 disk.sda1.out‚0$ memory_percent‚97.69301414489746' proc.memory.availableMB‚ 377.9765625" proc.memory.rssMB‚ 2796.6640625 proc.cpu.threads‚17' disk./.usagePercent‚39.300172274638 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚34058.263671875 disk.sda3.out‚ 1515.56640625 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.5361328125 disk.sda1.in‚ 8.759765625 cpu‚26.183138910273193 disk.dm-0.in‚ 25426.546875ÿ-²v4„6/ _wandbruntime‚15045  _runtime‚15045£' '…6ŠN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68447992 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2653516587008000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18529936783939 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684135 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2667234500980736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2653516604047360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚17549 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139141730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18758281426432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2672046523791939 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301684135 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162552396 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18758353219072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2672046881381301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162542405 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18530277333941 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚606274693314048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚606274602909184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843553865722966e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2667234518020096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684135 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2667234500980736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2653516604047360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚17549 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139141730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18758281426432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2672046523791939 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301684135 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162552396 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18758353219072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2672046881381301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162542405 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18530277333941 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚606274693314048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚606274602909184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843553865722966e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2667234518020096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68447992 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2653516587008000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71792640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18529936783939 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503503 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2674449328472064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139515700 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2693057605302784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139515698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2674449311432704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18608276830720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68635542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302513482 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30826241 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139515700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30826238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68635275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37809301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589154349262848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2674449311432704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162997782 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162987805 % _timestamp‚1.7843554259858983e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162987805 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚15084.679629164 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18607935491584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18607935491584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2679275152146853 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2679275509736103 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2693057246924288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589154330650624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18608276830720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚17599 _step‚1448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2674449328472064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2693057246924288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162997782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136+¡U (“6…P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2679275509736103 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2693057246924288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589154330650624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18608276830720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚17599 _step‚1448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2674449328472064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2693057246924288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162997782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139515698 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2660695609929728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚607962941574144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18808610923520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503503 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2674449328472064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139515700 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2693057605302784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139515698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2674449311432704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18608276830720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68635542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302513482 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30826241 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139515700 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30826238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68635275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37809301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589154349262848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2674449311432704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162997782 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162987805 % _timestamp‚1.7843554259858983e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162987805 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚15084.679629164 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18607935491584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18607935491584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2679275152146853¨ Ö*4”6/ _wandbruntime‚15085  _runtime‚15085¦}'•6‹N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163006848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2660842120601600 _step‚1449 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68639090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607997491602944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139523332 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162996857 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2693205858966528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607997382323712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18609289464832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18580893820849 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18609289464832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843554265993748e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18608948118528 trainer/global_step‚17599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37811376 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2660842137640960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30827986 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139523330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68639362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚589187759409152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139523332 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18809713581568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2674596569501696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2674596552462336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2679423031461809 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302520187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30827983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302530180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18608948118528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2679422673872447 _runtime‚15085.293134866 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18580553270847S À‚È–6N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18580893820849 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18609289464832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843554265993748e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18608948118528 trainer/global_step‚17599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37811376 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2660842137640960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30827986 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139523330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68639362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚589187759409152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139523332 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18809713581568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2674596569501696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2674596552462336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2679423031461809 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302520187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30827983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302530180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18608948118528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2679422673872447 _runtime‚15085.293134866 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18580553270847 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163006848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2660842120601600 _step‚1449 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68639090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607997491602944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139523332 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162996857 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2693205858966528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607997382323712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18609289464832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154© üÄ#¯4—6/ _wandbruntime‚15085  _runtime‚15085LrÄ> ˜6‡ 0loss/FRt1/assignment_loss‚1.4653408181629404 2loss/SRqqt1/assignment_loss‚0.4947658356174386 1loss/SRqqt1/detection_loss‚0.8236762475890673 /loss/FBt1/detection_loss‚0.2926355194617827 _runtime‚15085.296513154 /loss/FRt2/assignment_loss‚0.829705741602121 3loss/FBt2/assignment_loss‚0.0004906918993078138 .loss/FRt2/detection_loss‚1.240197428734073 &loss/total_loss‚7.1682585720516006 % _timestamp‚1.7843554266028411e+09 2loss/SRqqt2/assignment_loss‚0.4501865059385253 2loss/FBt1/assignment_loss‚0.018335146109050182 _step‚1450 /loss/FRt1/detection_loss‚0.5736435925068483 1loss/SRqqt2/detection_loss‚0.5768462998744692 /loss/FBt2/detection_loss‚0.4024347445559769 epoch‚43 trainer/global_step‚17599ízF?™6Œ 0loss/FRt1/assignment_loss‚1.4653408181629404 2loss/SRqqt1/assignment_loss‚0.4947658356174386 1loss/SRqqt1/detection_loss‚0.8236762475890673 /loss/FBt1/detection_loss‚0.2926355194617827 _runtime‚15085.296513154 /loss/FRt2/assignment_loss‚0.829705741602121 3loss/FBt2/assignment_loss‚0.0004906918993078138 .loss/FRt2/detection_loss‚1.240197428734073 &loss/total_loss‚7.1682585720516006 % _timestamp‚1.7843554266028411e+09 2loss/SRqqt2/assignment_loss‚0.4501865059385253 2loss/FBt1/assignment_loss‚0.018335146109050182 _step‚1450 /loss/FRt1/detection_loss‚0.5736435925068483 1loss/SRqqt2/detection_loss‚0.5768462998744692 /loss/FBt2/detection_loss‚0.4024347445559769 epoch‚43 trainer/global_step‚17599ª /”¡š6‚(:’ ç´ìÒ¤¦“ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï ¡14›6‚(:© ç´ìÒÝÌ—$ disk.nvme0c0n1.in‚25440.58203125 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1517.515625( proc.memory.availableMB‚ 373.05078125( proc.memory.percent‚17.0975923538208 network.sent‚ 2.8272117e+07$ disk./.usageGB‚585.6009101867676 disk.dm-1.out‚ 1517.515625& disk.nvme0n1.out‚34074.67626953125 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0# disk.dm-0.out‚34074.67626953125 disk.sda2.out‚ 170.53515625# proc.memory.rssMB‚ 2801.26953125 disk.dm-0.in‚ 25426.546875 disk.sda.out‚ 1688.05078125 cpu‚26.18300710047185 proc.cpu.threads‚17) disk./.usagePercent‚39.30016843452499 disk.rbd1.out‚ 47.7236328125 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.in‚ 2.25390625 network.recv‚ 4.026481e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚25440.58203125$ memory_percent‚97.72307872772217 disk.sda.in‚541.0478515625 disk.sda1.in‚ 8.759765625( disk.nvme0c0n1.out‚34074.67626953125ÅjûÑ4œ6/ _wandbruntime‚15102  _runtime‚15102ê3"yr=6ìz 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16541603207588196 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901328086853027 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037205442786216736 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.694206178188324 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6223254799842834 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.12635304033756256 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14396707713603973 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961466789245605 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7304660081863403 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 'particle/f_score‚0.6056533455848694 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 _step‚1451 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9806751012802124 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007004737854004 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04685671627521515 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 jet/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 (loss/val_total_loss‚7.71907332268292 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29827210307121277 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7311400175094604 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02747371606528759 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5315533876419067 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5086295008659363 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7163189053535461 )particle/sensitivity‚0.59554123878479 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15033742785453796 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.791408360004425 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37887197732925415 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6491839289665222 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6809065937995911 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7875353693962097 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8828765153884888 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02694319188594818 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7845239043235779 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21600519120693207 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2520308792591095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23536032438278198 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.38124701380729675 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6570029854774475 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6408286094665527 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 5 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24278524518013 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6290556788444519 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05236867442727089 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1812802106142044 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3656395375728607 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23934628069400787 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7729317545890808 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7632039189338684 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6414767503738403 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5682377815246582 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7859881520271301 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5691891312599182 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09037794172763824 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.875424861907959 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3020687699317932 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_1_of_1‚0.3487190008163452 ,jet/accuracy_3_of_4‚0.042860183864831924 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2393701821565628 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29789894819259644 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8454270958900452 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6554603576660156 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9455037117004395 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944208264350891 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8649281859397888 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7121581435203552 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34123119711875916 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2791707515716553 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6716066002845764 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2874749004840851 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03710507974028587 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03591939061880112 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027190566062927246 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21673649549484253 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.717584490776062 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.133630633354187 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33370015025138855 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_3‚0.5082651376724243 /particle/accuracy_4_of_4‚0.9479283690452576 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7209944128990173 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5474811792373657 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09845792502164841 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2875834107398987 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14839765429496765 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7915902733802795 ,jet/accuracy_3_of_3‚0.038866471499204636 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7125857472419739 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3184810280799866 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953998327255249 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28469252586364746 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8334612250328064 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7307250499725342 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6725713610649109 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7659323811531067 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24392834305763245 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28766751289367676 *jet/accuracy_2_of_4‚0.7720746397972107 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28766751289367676 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 particle/accuracy_1_of_5‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790241539478302 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25095322728157043 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12295933067798615 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10122077912092209 *jet/accuracy_1_of_3‚0.9282289147377014 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.630782961845398 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6408286094665527 particle/accuracy_3_of_3‚1 _runtime‚15102.464407808 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 epoch‚43 % _timestamp‚1.7843554437701483e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962919354438782 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026734009385108948 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955829381942749 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6152704954147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24096143245697021 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25339174270629883 +jet/accuracy_4_of_4‚0.01670061983168125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5637430548667908 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6596946120262146 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.875424861907959 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.8985283374786377 trainer/global_step‚17599 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917798638343811 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22672532498836517 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243221282958984 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6796282529830933 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7141140103340149 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6928619742393494ž=â©,ž6ñz 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28766751289367676 *jet/accuracy_2_of_4‚0.7720746397972107 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28766751289367676 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 particle/accuracy_1_of_5‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8537793159484863 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790241539478302 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25095322728157043 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12295933067798615 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10122077912092209 *jet/accuracy_1_of_3‚0.9282289147377014 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.630782961845398 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6408286094665527 particle/accuracy_3_of_3‚1 _runtime‚15102.464407808 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 epoch‚43 % _timestamp‚1.7843554437701483e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962919354438782 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026734009385108948 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955829381942749 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6152704954147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24096143245697021 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25339174270629883 +jet/accuracy_4_of_4‚0.01670061983168125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5637430548667908 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6596946120262146 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.875424861907959 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.8985283374786377 trainer/global_step‚17599 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917798638343811 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22672532498836517 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9243221282958984 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6796282529830933 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7141140103340149 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6928619742393494 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16541603207588196 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901328086853027 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037205442786216736 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.694206178188324 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6223254799842834 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_2_of_2‚0.12635304033756256 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14396707713603973 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961466789245605 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7304660081863403 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 'particle/f_score‚0.6056533455848694 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 _step‚1451 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9806751012802124 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007004737854004 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04685671627521515 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 jet/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 (loss/val_total_loss‚7.71907332268292 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29827210307121277 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7311400175094604 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02747371606528759 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5315533876419067 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5086295008659363 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7163189053535461 )particle/sensitivity‚0.59554123878479 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15033742785453796 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.791408360004425 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37887197732925415 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6491839289665222 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6809065937995911 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7875353693962097 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8828765153884888 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02694319188594818 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7845239043235779 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21600519120693207 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2520308792591095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23536032438278198 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.38124701380729675 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6570029854774475 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6408286094665527 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 5 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24278524518013 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6290556788444519 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05236867442727089 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1812802106142044 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3656395375728607 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23934628069400787 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7729317545890808 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7632039189338684 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6414767503738403 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5682377815246582 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7859881520271301 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5691891312599182 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09037794172763824 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.875424861907959 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3020687699317932 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_1_of_1‚0.3487190008163452 ,jet/accuracy_3_of_4‚0.042860183864831924 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2393701821565628 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29789894819259644 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8454270958900452 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6554603576660156 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5495833158493042 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9455037117004395 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944208264350891 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8649281859397888 9"Purity/*FRtþ+¼ Î1SRqqt1FBt/SRqqt_purity‚0.7121581435203552 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34123119711875916 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2791707515716553 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6716066002845764 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537793159484863 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2874749004840851 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03710507974028587 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03591939061880112 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027190566062927246 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21673649549484253 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.717584490776062 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.133630633354187 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33370015025138855 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_3‚0.5082651376724243 /particle/accuracy_4_of_4‚0.9479283690452576 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7209944128990173 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5474811792373657 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_3‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09845792502164841 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2875834107398987 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14839765429496765 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7915902733802795 ,jet/accuracy_3_of_3‚0.038866471499204636 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7125857472419739 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3184810280799866 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5115740299224854 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953998327255249 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28469252586364746 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8334612250328064 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9819256663322449 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7307250499725342 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6725713610649109 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7659323811531067 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24392834305763245 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505« 1°ŽÛ“Ÿ6‚(:ˆ ö´ìÒ¢¼º“ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿*òÅ. 6‚(:£ ö´ìÒð’• disk.dm-1.out‚ 1517.83984375$ memory_percent‚97.74069786071777 proc.memory.rssMB‚ 2804.21875) proc.memory.percent‚17.11559295654297 cpu‚26.18575642258832 disk.dm-0.in‚ 25546.609375 disk.sda2.out‚ 170.53515625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚34077.40966796875 disk.sda.in‚541.0478515625 disk.dm-1.in‚ 233.44140625 network.sent‚ 2.833552e+07 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.7236328125' proc.memory.availableMB‚ 370.1640625$ disk.nvme0c0n1.in‚25560.64453125# disk.dm-0.out‚34077.40966796875 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.sda.out‚1688.375" disk.nvme0n1.in‚25560.64453125$ disk./.usageGB‚585.6012420654297& disk.nvme0n1.out‚34077.40966796875 disk.sda3.out‚ 1517.83984375 disk.sda1.in‚ 8.759765625) disk./.usagePercent‚39.30019070718046 disk.sda2.in‚ 82.544921875 network.recv‚ 4.03084e+06 disk.rbd0.out‚0;U–“¡6‚(:ˆ …µìÒ‚×µ“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚235.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\%xš.¢6‚(:£ …µìÒñÜÞ• proc.memory.rssMB‚ 2804.21875( proc.memory.availableMB‚ 370.19921875$ disk.nvme0c0n1.in‚25560.64453125% disk.nvme0c0n1.out‚34080.61328125 disk.sda.out‚ 1688.9296875# disk.nvme0n1.out‚34080.61328125 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.sda2.in‚ 82.544921875$ memory_percent‚97.74048328399658) proc.memory.percent‚17.11559295654297 disk.dm-1.out‚ 1518.39453125 disk.drbd2165.out‚0" disk.nvme0n1.in‚25560.64453125 disk.rbd0.in‚ 0.16015625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.7861328125 proc.cpu.threads‚17 disk.dm-0.out‚34080.61328125 network.sent‚ 2.8392789e+07* disk./.usagePercent‚39.300192499233205$ disk./.usageGB‚585.6012687683105 disk.sda.in‚541.0478515625 cpu‚26.185306260044868 network.recv‚ 4.035147e+06 disk.sda1.in‚ 8.759765625 disk.sda2.out‚ 170.53515625 disk.sda1.out‚0 disk.sda3.out‚ 1518.39453125 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚ 25546.6093755u2‘£6‚(:† ”µìÒѸΓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0🭠2¤6‚(:§ ”µìÒïò• disk.sda3.in‚ 417.107421875$ disk./.usageGB‚586.3956184387207 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0" disk.dm-0.out‚34897.3486328125& proc.memory.availableMB‚ 370.203125 proc.memory.rssMB‚ 2804.21875 disk.drbd2165.out‚0" disk.nvme0n1.in‚25560.64453125% disk.nvme0n1.out‚34897.3486328125 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.7861328125$ memory_percent‚97.74045944213867 cpu‚26.184825651090826* disk./.usagePercent‚39.353501972118195 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1518.59765625 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1518.59765625$ disk.nvme0c0n1.in‚25560.64453125 network.recv‚ 4.038612e+06) proc.memory.percent‚17.11559295654297 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.dm-0.in‚ 25546.609375 disk.sda.in‚541.0478515625 disk.sda.out‚ 1689.1328125 disk.sda2.in‚ 82.544921875 disk.drbd2165.in‚ 2.25390625 network.sent‚ 2.8397828e+07 disk.rbd0.out‚0' disk.nvme0c0n1.out‚34897.3486328125ÛA4¥6/ _wandbruntime‚15141  _runtime‚15141\…–áʦ6Ä ' lr-AdamW/pg2‚1.5754634306337837e-05 trainer/global_step‚17649 % _timestamp‚1.7843554832604089e+09 _runtime‚ 15141.9539665 _step‚1452 ' lr-AdamW/pg1‚1.5754634306337837e-05{Ÿô"ϧ6É _runtime‚ 15141.9539665 _step‚1452 ' lr-AdamW/pg1‚1.5754634306337837e-05 ' lr-AdamW/pg2‚1.5754634306337837e-05 trainer/global_step‚17649 % _timestamp‚1.7843554832604089e+09¬ [/$«4¨6/ _wandbruntime‚15141  _runtime‚15141a•q’(©6P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2702417652142080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303540522 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2702417293763584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18648218979152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702417293763584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140017964 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843554832614484e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30916008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140017964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30916011 trainer/global_step‚17649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163522556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18676705538048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68860397 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18647878429262 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702417652142080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68860678 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2669929806737408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚609719660363776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37944667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37944389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590841699289088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2683740929564672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140017966 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163512579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590841680676864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18877961074688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702417293763584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140017964 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843554832614484e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30916008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140017964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30916011 trainer/global_step‚17649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163522556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18676705538048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68860397 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18647878429262 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702417652142080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68860678 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2669929806737408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚609719660363776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37944667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37944389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590841699289088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2683740929564672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140017966 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163512579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590841680676864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18877961074688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557220 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18648889482984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2683888170594304 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163531622 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18677718172160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140025596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590875109435392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small*{ÖtÔ_pool.peak‚269 trainer/global_step‚17649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163521631 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68864240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547227 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670076334448640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303547227 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163531622 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140025598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590875128047616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140025598 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18677718172160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2688725206892264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163521631 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30917756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30917753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37946487 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18878984490496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2688725564481626 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843554838749127e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18649230032986 _step‚1454 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2683888187633664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702565547420160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2683888170594304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702565905805824`€?ø'­6N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303557220 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557220 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18648889482984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2683888170594304 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163531622 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18677718172160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140025596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590875109435392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚17649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163521631 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68864240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547227 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670076334448640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303547227 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163531622 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140025598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590875128047616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140025598 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18677718172160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2688725206892264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163521631 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30917756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30917753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37946487 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18878984490496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2688725564481626 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843554838749127e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18649230032986 _step‚1454 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2683888187633664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702565547420160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2683888170594304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702565905805824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2683888187633664® «ß&˜4®6/ _wandbruntime‚15142  _runtime‚15142nE6y ¯6„ /loss/FRt1/assignment_loss‚1.430456891049384 1loss/FBt1/assignment_loss‚1.0866975048749e-05 /loss/FBt2/detection_loss‚0.3799573752124477 trainer/global_step‚17649 $ _timestamp‚1.784355483878093e+09 _step‚1455 2loss/SRqqt2/assignment_loss‚0.4090557616360695 epoch‚44 0loss/FRt2/assignment_loss‚0.7605711992222575 2loss/SRqqt1/assignment_loss‚0.4663477213210942 1loss/SRqqt2/detection_loss‚0.5571317856968543 /loss/FBt1/detection_loss‚0.2592362678315978 %loss/total_loss‚6.821573482724508 _runtime‚15142.571424067 3loss/FBt2/assignment_loss‚0.0071004854160632465 /loss/FRt1/detection_loss‚0.5108344380557612 .loss/FRt2/detection_loss‚1.164972289556739 1loss/SRqqt1/detection_loss‚0.87589840075119177‘°6‰ epoch‚44 0loss/FRt2/assignment_loss‚0.7605711992222575 2loss/SRqqt1/assignment_loss‚0.4663477213210942 1loss/SRqqt2/detection_loss‚0.5571317856968543 /loss/FBt1/detection_loss‚0.2592362678315978 %loss/total_loss‚6.821573482724508 _runtime‚15142.571424067 3loss/FBt2/assignment_loss‚0.0071004854160632465 /loss/FRt1/detection_loss‚0.5108344380557612 .loss/FRt2/detection_loss‚1.164972289556739 1loss/SRqqt1/detection_loss‚0.8758984007511917 /loss/FRt1/assignment_loss‚1.430456891049384 1loss/FBt1/assignment_loss‚1.0866975048749e-05 /loss/FBt2/detection_loss‚0.3799573752124477 trainer/global_step‚17649 $ _timestamp‚1.784355483878093e+09 _step‚1455 2loss/SRqqt2/assignment_loss‚0.4090557616360695¯ Ðs²{“±6‚(:ˆ £µìÒᜧ“ gpu.0.gpu‚40.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kø…2²6‚(:§ £µìÒ¬¥³– disk.rbd0.out‚0 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 47.8486328125) proc.memory.percent‚17.11559295654297 disk.sda1.out‚0$ memory_percent‚97.74212837219238' proc.memory.availableMB‚ 369.9296875 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚ 25546.609375# disk.dm-0.out‚34902.88134765625 disk.rbd0.in‚ 0.16015625 network.recv‚ 4.042145e+06$ disk./.usageGB‚586.3687705993652 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1518.984375) disk./.usagePercent‚39.35170019109221( disk.nvme0c0n1.out‚34902.88134765625" disk.nvme0n1.in‚25560.64453125 proc.memory.rssMB‚ 2804.21875 cpu‚26.184394168358217 network.sent‚ 2.8402912e+07 disk.sda.out‚ 1689.51953125 disk.sda2.in‚ 82.544921875 disk.dm-1.out‚ 1518.984375 proc.cpu.threads‚17& disk.nvme0n1.out‚34902.88134765625 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625$ disk.nvme0c0n1.in‚25560.64453125 üõ¼“³6‚(:ˆ ²µìÒ‰®¢• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÎSu2´6‚(:§ ²µìÒóîø— disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 369.93359375$ disk./.usageGB‚585.6547927856445 disk.sda3.out‚ 1519.4453125 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚25660.16796875" disk.nvme0n1.in‚25674.19140625 proc.cpu.threads‚17 disk.dm-1.out‚ 1519.4453125) disk./.usagePercent‚39.30378454094613# disk.nvme0c0n1.in‚ 25674.1484375$ memory_percent‚97.74210453033447 proc.memory.rssMB‚ 2804.21875 disk.sda1.out‚0 disk.sda.out‚ 1689.98046875 disk.sda2.in‚ 82.544921875) proc.memory.percent‚17.11559295654297 cpu‚26.18392679582077 network.sent‚ 2.8465946e+07 network.recv‚ 4.046585e+06 disk.sda3.in‚ 417.107421875% disk.nvme0n1.out‚34906.2216796875 disk.sda1.in‚ 8.759765625" disk.dm-0.out‚34906.2216796875 disk.sda2.out‚ 170.53515625 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625' disk.nvme0c0n1.out‚34906.2216796875 disk.sda.in‚541.0478515625 disk.rbd1.out‚ 47.8486328125ェ(+µ6‚(:  ÁµìÒøÐ’–( proc.memory.availableMB‚ 370.19140625 disk.rbd1.in‚ 2.1796875 disk.sda.in‚541.0478515625# memory_percent‚97.7405309677124 network.recv‚ 4.050296e+06 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 47.9111328125$ disk.nvme0n1.out‚34911.240234375 disk.sda3.out‚ 1519.78125& disk.nvme0c0n1.out‚34911.240234375 disk.dm-0.in‚ 25699.3671875! disk.dm-0.out‚34911.240234375 disk.sda.out‚ 1690.31640625) proc.memory.percent‚17.11559295654297$ disk.nvme0c0n1.in‚25713.40234375 disk.sda2.out‚ 170.53515625 cpu‚26.183423094030076) disk./.usagePercent‚39.30378300490093 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 proc.cpu.threads‚17 network.sent‚ 2.8470463e+07$ disk./.usageGB‚585.6547698974609 disk.sda1.out‚0 disk.dm-1.out‚ 1519.78125 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2804.21875 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚25713.40234375\r§ë“¶6‚(:ˆ ÁµìÒÞ¦ö” gpu.0.gpu‚96.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K‡î 4·6/ _wandbruntime‚15181  _runtime‚15181ž2Áw̸6Æ _runtime‚15181.789836495 _step‚1456 ' lr-AdamW/pg1‚1.5099170205625866e-05 ' lr-AdamW/pg2‚1.5099170205625866e-05 trainer/global_step‚17699 % _timestamp‚1.7843555230962543e+09Æ—s;ѹ6Ë ' lr-AdamW/pg2‚1.5099170205625866e-05 trainer/global_step‚17699 % _timestamp‚1.7843555230962543e+09 _runtime‚15181.789836495 _step‚1456 ' lr-AdamW/pg1‚1.5099170205625866e-05° Ë5‹¸4º6/ _wandbruntime‚15181  _runtime‚15181§ÃmT #»6€P ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677255340331008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18698494916170 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2691102998085632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69052296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31003261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2691102981046272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163967031 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163977008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695954192836428 _runtime‚15181.794357562 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2691102981046272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18727059700736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31003258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69052577 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140399564 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18929229988864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38049038 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2709830399125504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18727401039872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304366595 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2691102998085632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611442367204352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2709830040747008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38049316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚592513118603264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18929139314688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140399566 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163977008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140399564 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚17699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304376574 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18727401039872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2709830399125504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2677255357370368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_po‰•`vùol.freed‚ 163967031 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2709830040747008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18727059700736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2695953835247178 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611442257917952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496b–.Ú (¼6…P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38049038 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2709830399125504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18727401039872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304366595 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2691102998085632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611442367204352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2709830040747008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38049316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚592513118603264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18929139314688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140399566 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163977008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140399564 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚17699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304376574 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18727401039872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2709830399125504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2677255357370368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163967031 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2709830040747008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18727059700736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2695953835247178 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611442257917952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677255340331008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18698494916170 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2691102998085632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69052296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31003261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2691102981046272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163967031 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163977008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695954192836428 _runtime‚15181.794357562 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2691102981046272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18727059700736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31003258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69052577 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140399564 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18929229988864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144± ¨6·A4½6/ _wandbruntime‚15182  _runtime‚15182Jtæ'¾6‹N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140407196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611476697631232 _runtime‚15182.404554075 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163976083 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2691250222075904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31005003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18930150269440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843555237107768e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163986074 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383279 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592546565974016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611476813201920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18728072327680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2691250239115264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18728413673984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304393272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592546547361792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2696101356972772 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69056419 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚17699 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163976083 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69056158 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163986074 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18728072327680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677401868042240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18699505969892 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38051413 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18699846519894 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2691250222075904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18728413673984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140407196 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677401851002880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2691250239115264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18930247227904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1458 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1æ=ü~'¿6N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31005003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18930150269440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843555237107768e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163986074 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383279 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592546565974016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611476813201920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18728072327680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2691250239115264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18728413673984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304393272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592546547361792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2696101356972772 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69056419 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚17699 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163976083 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69056158 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163986074 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18728072327680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677401868042240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18699505969892 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38051413 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18699846519894 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2691250222075904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18728413673984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140407196 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677401851002880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2691250239115264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18930247227904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1458 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140407196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611476697631232 _runtime‚15182.404554075 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163976083 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2691250222075904² É›4À6/ _wandbruntime‚15182  _runtime‚15182ˆj³âÁ6Œ 4loss/FBt2/assignment_loss‚4.0032933677831525e-08 .loss/FRt2/detection_loss‚1.159586201518286 3loss/SRqqt1/assignment_loss‚0.37758939598703994 /loss/FRt1/detection_loss‚0.5263110258433727 /loss/FBt1/detection_loss‚0.3159688013625922 trainer/global_step‚17699 % _timestamp‚1.7843555237141805e+09 _step‚1459 0loss/FBt2/detection_loss‚0.38106642434287546 _runtime‚15182.407463458 0loss/FRt2/assignment_loss‚0.7094465574287359 3loss/FBt1/assignment_loss‚5.054927098440815e-05 1loss/SRqqt1/detection_loss‚0.8272823352366686 1loss/SRqqt2/detection_loss‚0.5377885604366852 %loss/total_loss‚6.851093580448657 epoch‚44 0loss/FRt1/assignment_loss‚1.5253749001994092 3loss/SRqqt2/assignment_loss‚0.49062878878907323õÚï#Â6‘ trainer/global_step‚17699 % _timestamp‚1.7843555237141805e+09 _step‚1459 0loss/FBt2/detection_loss‚0.38106642434287546 _runtime‚15182.407463458 0loss/FRt2/assignment_loss‚0.7094465574287359 3loss/FBt1/assignment_loss‚5.054927098440815e-05 1loss/SRqqt1/M@Gôdetection_loss‚0.8272823352366686 1loss/SRqqt2/detection_loss‚0.5377885604366852 %loss/total_loss‚6.851093580448657 epoch‚44 0loss/FRt1/assignment_loss‚1.5253749001994092 3loss/SRqqt2/assignment_loss‚0.49062878878907323 4loss/FBt2/assignment_loss‚4.0032933677831525e-08 .loss/FRt2/detection_loss‚1.159586201518286 3loss/SRqqt1/assignment_loss‚0.37758939598703994 /loss/FRt1/detection_loss‚0.5263110258433727 /loss/FBt1/detection_loss‚0.3159688013625922³ P¾œ¥“Ã6‚(:ˆ еìÒÆë“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.216' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥€6Ä6‚(:« еìÒϼɕ disk.dm-1.out‚ 1520.1484375' proc.memory.availableMB‚ 369.7265625 disk.sda1.in‚ 8.759765625 network.sent‚ 2.8475654e+07 disk.dm-0.in‚ 25699.3671875& disk.nvme0n1.out‚34914.56591796875 disk.rbd0.out‚0* disk./.usagePercent‚39.301364501724954 disk.drbd2165.in‚ 2.25390625 disk.sda1.out‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚34914.56591796875 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625 cpu‚26.182981731822306$ disk.nvme0c0n1.in‚25713.40234375 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1520.1484375$ memory_percent‚97.74336814880371 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0# disk.dm-0.out‚34914.56591796875 disk.sda.out‚ 1690.68359375 disk.rbd1.out‚ 47.9111328125 network.recv‚ 4.053617e+06 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚25713.40234375 disk.rbd1.in‚ 2.1796875) proc.memory.percent‚17.11559295654297$ disk./.usageGB‚585.6187324523926 proc.memory.rssMB‚ 2804.21875@AŽ“Å6‚(:ˆ ßµìÒù¤“” gpu.0.gpu‚97.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êY¥2Æ6‚(:§ ßµìÒÜñÀ– disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚25713.40234375 disk.sda3.in‚ 417.107421875 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1520.63671875 disk.sda2.in‚ 82.544921875 disk.sda1.in‚ 8.759765625 disk.dm-0.in‚ 25699.3671875 disk.sda3.out‚ 1520.63671875 proc.memory.rssMB‚ 2804.21875) disk./.usagePercent‚39.30135451743112" disk.dm-0.out‚34920.4462890625 disk.sda2.out‚ 170.53515625% disk.nvme0n1.out‚34920.4462890625 network.sent‚ 2.8539231e+07 disk.rbd0.in‚ 0.16015625 disk.sda.in‚541.0478515625 disk.drbd2165.out‚0$ memory_percent‚97.74491786956787 proc.cpu.threads‚17' disk.nvme0c0n1.out‚34920.4462890625$ disk./.usageGB‚585.6185836791992 disk.sda.out‚ 1691.171875) proc.memory.percent‚17.11559295654297 network.recv‚ 4.058292e+06" disk.nvme0n1.in‚25713.40234375 cpu‚26.18251934869791 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 48.0361328125( proc.memory.availableMB‚ 369.47265625 disk.rbd0.out‚0 disk.sda1.out‚0Ä}{4Ç6/ _wandbruntime‚15221  _runtime‚15221ãkWËÈ6Å _runtime‚15221.839694027 _step‚1460 ' lr-AdamW/pg1‚1.4457157999855755e-05 ' lr-AdamW/pg2‚1.4457157999855755e-05 trainer/global_step‚17749 $ _timestamp‚1.784355563146172e+09±»±ÐÉ6Ê ' lr-AdamW/pg1‚1.4457157999855755e-05 ' lr-AdamW/pg2‚1.4457157999855755e-05 trainer/global_step‚17749 $ _timestamp‚1.784355563146172e+09 _runtime‚15221.839694027 _step‚1460´ D°È4Ê6/ _wandbruntime‚15221  _runtime‚15221«ú•(Ë6P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164421483 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69243717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613164922826752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2703329985327686 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38153484 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18778096541696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2698465032527872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613165030016000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2698465032527872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18980454874112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140781164 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2717242787730432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18749111403078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2698465049567232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202647 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31090508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164421483 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18777755202560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18749451952968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2698465049567232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38153209 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2717242787730432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2717243146108928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18980366297088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164431460 _step‚1461 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18777755202560 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2717243146108928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594184556529664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2684580890963968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2684580873924608 % _timestamp‚1.7843555631472192e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140781166 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305212626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164431460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 _runtime‚15221.840915884 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2717243146108928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚594184556529664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2684580890963968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2684580873924608 % _timestamp‚1.7843555631472192e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140781166 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305212626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164431460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 _runtime‚15221.840915884 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2698465032527872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613165030016000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2698465032527872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18980454874112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140781164 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2717242787730432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18749111403078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2698465049567232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202647 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31090508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164421483 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18777755202560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18749451952968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2698465049567232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38153209 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2717242787730432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2717243146108928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18980366297088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164431460 _step‚1461 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18777755202560 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18981391907328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140788798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164430535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140788796 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594218003900416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69247528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38155275 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31092256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594217985288192 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2684727384596480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38155537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843555637612607e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18778767829504 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2698612290596864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2703477864642642 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2717391399772672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2698612290596864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1462 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38155537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843555637612607e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18778767829504 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2698612290596864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2703477864642642 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2717391399772672 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2698612290596864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1462 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18981391907328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140788798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164430535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140788796 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594218003900416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69247528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38155275 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31092256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594217985288192 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2684727384596480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2691906424557568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843556032433813e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164885912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚595856013068288 trainer/global_step‚17799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2724655893092352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614887706357248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2705827101048832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724655534713856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2705827084009472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2710706492997444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚1465 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2705827084009472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614887613848064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69434840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141162766 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164875935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38256833 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306038699 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚15261.937067436 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2691906424557568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843556032433813e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164885912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚595856013068288 trainer/global_step‚17799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2724655893092352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614887706357248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2705827101048832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724655534713856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2705827084009472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2710706492997444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚1465 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2705827084009472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614887613848064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69434840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141162766 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164875935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38256833 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306038699 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚15261.937067436 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065376 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164894978 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1466 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69438396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724804146756096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19032727297024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2710853657133788 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18801079493710 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164884987 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31179506 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 $ _timestamp‚1.784355603856054e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18829463331328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38259146 _runtime‚15262.549853169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2705974342078464 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141170398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306055383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614922169123840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2705974342078464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141170396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164894978 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164884987 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18829804677632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2710854014723150 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18829463331328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306055383 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614922061941760 trainer/global_step‚17799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19032638727168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2692052935229440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306065376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚595889441826816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595889423214592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18829804677632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69438652 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141170396 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2692052918190080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31179503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1’Ž~]'á6N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2710853657133788 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18801079493710 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164884987 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31179506 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 $ _timestamp‚1.784355603856054e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18829463331328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38259146 _runtime‚15262.549853169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2705974342078464 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141170398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306055383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614922169123840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2705974342078464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141170396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164894978 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164884987 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18829804677632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2710854014723150 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18829463331328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306055383 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614922061941760 trainer/global_step‚17799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19032638727168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2692052935229440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306065376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚595889441826816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595889423214592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18829804677632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69438652 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141170396 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2692052918190080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31179503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2705974325039104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141170398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38258893 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065376 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164894978 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1466 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69438396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724804146756096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19032727297024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2º 2µhÞ4â6/ _wandbruntime‚15262  _runtime‚152624ú÷ ã6† /loss/FRt1/detection_loss‚0.5421599781892356 /loss/FRt2/detection_loss‚1.2094498353920595 1loss/SRqqt1/detection_loss‚0.8809285204422812 .loss/SRqqt2/detection_loss‚0.5620862887622 0loss/FRt2/assignment_loss‚0.6801296488784166 3loss/SRqqt2/assignment_loss‚0.49309972531018226 2loss/FBt1/assignment_loss‚0.007829787939357164 2loss/FBt2/assignment_loss‚5.70058801191236e-05 0loss/FRt1/assignment_loss‚1.3687226493776097 /loss/FBt1/detection_loss‚0.6552879448621146 %loss/total_loss‚7.300629764427008 epoch‚44 _runtime‚15262.553223643 2loss/SRqqt1/assignment_loss‚0.4438050818481882 0loss/FBt2/detection_loss‚0.45707329754524473 trainer/global_step‚17799 % _timestamp‚1.7843556038598971e+09 _step‚1467[0ðæä6‹ 2loss/SRqqt1/assignment_loss‚0.4438050818481882 0loss/FBt2/detection_loss‚0.45707329754524473 trainer/global_step‚17799 % _timestamp‚1.7843556038598971e+09 _step‚1467 /loss/FRt1/detection_loss‚0.5421599781892356 /loss/FRt2/detection_loss‚1.2094498353920595 1loss/SRqqt1/detection_loss‚0.8809285204422812 .loss/SRqqt2/detection_loss‚0.5620862887622 0loss/FRt2/assignment_loss‚0.6801296488784166 3loss/SRqqt2/assignment_loss‚0.49309972531018226 2loss/FBt1/assignment_loss‚0.007829787939357164 2loss/FBt2/assignment_loss‚5.70058801191236e-05 0loss/FRt1/assignment_loss‚1.3687226493776097 /loss/FBt1/detection_loss‚0.6552879448621146 %loss/total_loss‚7.300629764427008 epoch‚44 _runtime‚15262.553223643» ÚüãÆœå6‚(:‘ ›¶ìÒæ¡Ò• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³³Ê73æ6‚(:¨ ›¶ìҜݘ— disk.sda1.out‚0! disk.nvme0n1.in‚ 25817.9140625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0$ memory_percent‚97.74370193481445 proc.cpu.threads‚17& disk.nvme0n1.out‚34935.51611328125 disk.sda3.out‚ 1521.9765625 disk.rbd1.in‚ 2.1796875& proc.memory.availableMB‚ 369.671875# disk.nvme0c0n1.in‚ 25817.9140625( disk.nvme0c0n1.out‚34935.51611328125 disk.sda.out‚ 1692.51171875 disk.sda2.in‚ 82.544921875 disk.rbd0.out‚0) disk./.usagePercent‚39.30188010089892 disk.dm-1.out‚ 1521.9765625 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 48.1611328125 network.sent‚ 2.8617576e+07 network.recv‚ 4.072534e+06$ disk./.usageGB‚585.6264152526855 disk.sda1.in‚ 8.759765625 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625 proc.memory.rssMB‚ 2804.21875 cpu‚26.180673548211388 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚25803.87890625# disk.dm-0.out‚34935.51611328125 disk.sda.in‚541.0478515625) proc.memory.percent‚17.11559295654297.]èžç6‚(:“ ª¶ìÒÛ‰À“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»É¯2è6‚(:§ ª¶ìÒÌɺ– disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 25817.9140625# disk./.usageGB‚585.626392364502 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625& disk.nvme0n1.out‚34938.57177734375 proc.memory.rssMB‚ 2804.21875 cpu‚26.180172676380224* disk./.usagePercent‚39.301878564853716 disk.dm-0.in‚25803.87890625) proc.memory.percent‚17.11559295654297 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1522.32421875 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.74484634399414 network.recv‚ 4.07635e+06# disk.dm-0.out‚34938.57177734375 disk.sda.in‚541.0478515625 disk.rbd1.out‚ 48.1611328125 disk.sda.out‚ 1692.859375 disk.sda2.in‚ 82.544921875 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625! disk.nvme0n1.in‚ 25817.9140625 network.sent‚ 2.8680084e+07 disk.sda2.out‚ 170.53515625( disk.nvme0c0n1.out‚34938.57177734375 proc.cpu.threads‚17 disk.dm-1.out‚ 1522.32421875 disk.drbd2165.out‚0& proc.memory.availableMB‚ 369.484375üäQ“é6‚(:ˆ ¹¶ìÒÑÆå“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿéh›3ê6‚(:¨ ¹¶ìÒŽ¨Ý•# disk.dm-0.out‚34944.30810546875 disk.rbd1.in‚ 2.1796875 disk.dm-1.in‚ 233.44140625 network.sent‚ 2.8685723e+07 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 48.2236328125 disk.dm-1.out‚ 1522.8046875 disk.drbd2165.out‚0$ memory_percent‚97.74484634399414 proc.cpu.threads‚17 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1522.8046875) proc.memory.percent‚17.11559295654297) disk./.usagePercent‚39.30194000666193( disk.nvme0c0n1.out‚34944.30810546875 disk.drbd2165.in‚ 2.25390625& proc.memory.availableMB‚ 369.484375 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 25937.8046875 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.dm-0.in‚25923.76953125& disk.nvme0n1.out‚34944.30810546875# disk.nvme0c0n1.in‚ 25937.8046875 cpu‚26.179738000455938 network.recv‚ 4.080129e+06 disk.sda.out‚ 1693.33984375$ disk./.usageGB‚585.6273078918457 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 proc.memory.rssMB‚ 2804.21875Àk¯4ë6/ _wandbruntime‚15301  _runtime‚15301¬þq^Æì6Æ ' lr-AdamW/pg1‚1.3213653347479866e-05 ' lr-AdamW/pg2‚1.3213653347479866e-05 trainer/global_step‚17849 % _timestamp‚1.7843556431299338e+09 _runtime‚15301.823493287 _step‚ku1468³ñ ¢Ñí6Ë _runtime‚15301.823493287 _step‚1468 ' lr-AdamW/pg1‚1.3213653347479866e-05 ' lr-AdamW/pg2‚1.3213653347479866e-05 trainer/global_step‚17849 % _timestamp‚1.7843556431299338e+09¼ Z>û4î6/ _wandbruntime‚15301  _runtime‚15301Lc´(ï6P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18879146206208 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597527450994688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38360442 _step‚1469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2713189135491072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165330387 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2732068640075776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2699231941111808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38360175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141544364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19082929178624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2713189135491072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306884730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31265011 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2732068640075776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18879487545344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18879146206208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141544366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2699231958151168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306874751 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19082836407296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843556431310925e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165340364 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2732068281697280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141544364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31265008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597527432382464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2732068281697280 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2713189152530432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165340364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616610268789760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38360175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141544364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19082929178624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2713189135491072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306884730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31265011 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2732068640075776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18879487545344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18879146206208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141544366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2699231958151168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306874751 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19082836407296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843556431310925e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165340364 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2732068281697280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141544364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31265008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597527432382464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2732068281697280 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2713189152530432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165340364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616610268789760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38360442 _step‚1469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2713189135491072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165330387 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2732068640075776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2699231941111808½ ØÛ´ð4ñ6/ _wandbruntime‚15302  _runtime‚15302@°ÖÁ'ò6‹N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141551998 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2699378451783680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69628970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38362470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2718229807214296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306891435 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18880500179456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31266753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2713336376520704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2718230164803658 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141551996 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚15302.437627906 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚17849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165339439 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616644832343552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18880500179456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306891435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚597560879753216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19083952590336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165349430 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19083866117632 % _timestamp‚1.7843556437439148e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2732216535353856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31266756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38362217 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169µP}eó6N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165339439 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616644832343552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18880500179456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306891435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚597560879753216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19083952590336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165349430 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19083866117632 % _timestamp‚1.7843556437439148e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2732216535353856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31266756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38362217 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141551998 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2699378451783680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69628970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38362470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2718229807214296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306891435 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18880500179456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31266753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2713336376520704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2718230164803658 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141551996 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚15302.437627906 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚17849¾ …ꓤ4ô6/ _wandbruntime‚15302  _runtime‚15302´ô~õ6ˆ 1loss/SRqqt2/detection_loss‚0.5858446965807881 " _timestamp‚1.7843556437481e+09 _runtime‚15302.441408394 3loss/SRqqt2/assignment_loss‚0.41457616819520127 1loss/SRqqt1/detection_loss‚0.8542468324226274 %loss/total_loss‚7.437718169547718 2loss/FBt1/assignment_loss‚0.000378881761239151 /loss/FBt1/detection_loss‚0.7807545975220445 trainer/global_step‚17849 _step‚1471 0loss/FRt2/assignment_loss‚0.7702536713367689 4loss/FBt2/assignment_loss‚3.0054535744616917e-05 /loss/FBt2/detection_loss‚0.4743221805642368 epoch‚44 0loss/FRt1/assignment_loss‚1.4931599815600067 3loss/SRqqt1/assignment_loss‚0.40219472691178515 /loss/FRt1/detection_loss‚0.5418546903640481 /loss/FRt2/detection_loss‚1.1201016877932275fÍuö6 1loss/SRqqt1/detection_loss‚0.8542468324226274 %loss/total_loss‚7.437718169547718 2loss/FBt1/assignment_loss‚0.000378881761239151 /loss/FBt1/detection_loss‚0.7807545975220445 trainer/global_step‚17849 _step‚1471 0loss/FRt2/assignment_loss‚0.7702536713367689 4loss/FBt2/assignment_loss‚3.0054535744616917e-05 /loss/FBt2/detection_loss‚0.4743221805642368 epoch‚44 0loss/FRt1/assignment_loss‚1.4931599815600067 3loss/SRqqt1/assignment_loss‚0.40219472691178515 /loss/FRt1/detection_loss‚0.5418546903640481 /loss/FRt2/detection_loss‚1.1201016877932275 1loss/SRqqt2/detection_loss‚0.5858446965807881 " _timestamp‚1.7843556437481e+09 _runtime‚15302.441408394 3loss/SRqqt2/assignment_loss‚0.41457616819520127¿ â§Yö“÷6‚(:ˆ ȶìÒ¹­»– gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÝfA,ø6‚(:¡ ȶìÒÖîÝ— proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 25937.8046875( disk./.usagePercent‚39.3019487109181# disk.nvme0c0n1.in‚ 25937.8046875 disk.dm-0.in‚25923.76953125 disk.sda2.in‚ 82.544921875( proc.memory.availableMB‚ 369.47265625$ disk./.usageGB‚585.6274375915527 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 network.recv‚ 4.083402e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 1523.078125 network.sent‚ 2.8690337e+07 disk.dm-1.in‚ 233.44140625! disk.dm-0.out‚34947.853515625 proc.memory.rssMB‚ 2804.21875 disk.sda1.out‚0& disk.nvme0c0n1.out‚34947.853515625$ disk.nvme0n1.out‚34947.853515625 disk.sda.in‚541.0478515625) proc.memory.percent‚17.11559295654297 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 48.2236328125 disk.sda.out‚ 1693.61328125 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1523.078125$ memory_percent‚97.74491786956787 cpu‚26.179294492481734º°מù6‚(:“ ׶ìÒò´Å“ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Öc,4ú6‚(:© ׶ì񢔥 network.sent‚ 2.8753582e+07 network.recv‚ 4.087854e+06 disk.sda.in‚541.0478515625 disk.rbd1.out‚ 48.2861328125( disk.nvme0c0n1.out‚34953.65576171875 disk.sda3.out‚ 1523.4921875 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0( proc.memory.availableMB‚ 369.49609375) proc.memory.percent‚17.11559295654297 cpu‚26.17883301961765# disk.nvme0c0n1.in‚ 25937.8046875# disk.dm-0.out‚34953.65576171875 proc.cpu.threads‚17 disk.sda.out‚ 1694.02734375 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 25937.8046875 disk.dm-1.in‚ 233.44140625) disk./.usagePercent‚39.30194973494823 disk.drbd2165.out‚0 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625$ disk./.usageGB‚585.6274528503418 disk.rbd1.in‚ 2.1796875 disk.dm-0.in‚25923.76953125 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚34953.65576171875$ memory_percent‚97.74477481842041 proc.memory.rssMB‚ 2804.21875 disk.drbd2165.in‚ 2.25390625 disk.dm-1.out‚ 1523.4921875?< ‚4û6/ _wandbruntime‚15341  _runtime‚15341MÜó˜Ìü6Æ _runtime‚15341.816841646 _step‚1472 ' lr-AdamW/pg1‚1.2612240768854182e-05 ' lr-AdamW/pg2‚1.2612240768854182e-05 trainer/global_step‚17899 % _timestamp‚1.7843556831232715e+09M, +Ñý6Ë ' lr-AdamW/pg1‚1.2612240768854182e-05 ' lr-AdamW/pg2‚1.2612240768854182e-05 trainer/global_step‚17899 % _timestamp‚1.7843556831232715e+09 _runtime‚15341.816841646 _step‚1472À b Ö4þ6/ _wandbruntime‚15341  _runtime‚153416[î@(ÿ6P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2739481028680704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18929841708032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2739481028680704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚17899 _runtime‚15341.821331111 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18929841708032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38463002 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2706557491744768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2720551186972672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720782 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2725458435569210 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165784839 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _step‚1473 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599198870308864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141925964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165794816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18930183047168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19133973897216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69815260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618332961881088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141925966 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900960863802 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2720551204012032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599198888921088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101ŠÎû#€7†P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2720551204012032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18901301413692 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165784839 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141925966 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31352261 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2739481387059200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843556831243436e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141925964 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2739481387059200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141925966 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900960863802 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2720551204012032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599198888921088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2739481028680704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18929841708032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2739481028680704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚17899 _runtime‚15341.821331111 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18929841708032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38463002 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2706557491744768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2720551186972672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720782 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2725458435569210 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165784839 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _step‚1473 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599198870308864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141925964 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165794816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18930183047168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19133973897216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_twˆ*¨rain_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69815260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618332961881088 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69819070 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2739629282337280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2720698428002304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18931195681280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165803882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38465067 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141933598 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307727487 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19135097016320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚17899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2739629640722944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720698428002304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1474 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38465335 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚15342.431463341 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307727487 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843556837375462e+09 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991ö9î 'ƒ7N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307727487 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 % _timestamp‚1.7843556837375462e+09 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165793891 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69819070 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2739629282337280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2720698428002304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18931195681280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165803882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38465067 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141933598 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307727487 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19135097016320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚17899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2739629640722944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720698428002304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1474 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38465335 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚15342.431463341 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ‘ £q4„7/ _wandbruntime‚15342  _runtime‚15342‘ðP …7Ž 3loss/SRqqt1/assignment_loss‚0.45224249916676484 3loss/SRqqt2/assignment_loss‚0.44450941177552467 %loss/total_loss‚7.190428279785653 epoch‚44 % _timestamp‚1.7843556837409873e+09 0loss/FRt1/assignment_loss‚1.4593659042698857 4loss/FBt1/assignment_loss‚5.5321498670145454e-05 /loss/FRt2/detection_loss‚1.2272715713430846 /loss/FBt1/detection_loss‚0.2892855174930042 3loss/FBt2/assignment_loss‚7.620975062489303e-05 1loss/SRqqt1/detection_loss‚0.9143437478756695 0loss/FBt2/detection_loss‚0.48976783464346835 trainer/global_step‚17899 _step‚1475 0loss/FRt2/assignment_loss‚0.8587109728338329 0loss/FRt1/detection_loss‚0.48276093374982154 1loss/SRqqt2/detection_loss‚0.5720383553853026 _runtime‚15342.434314373ÐnE†7“ 3loss/FBt2/assignment_loss‚7.620975062489303e-05 1loss/SRqqt1/detection_loss‚0.9143437478756695 0loss/FBt2/detection_loss‚0.48976783464346835 trainer/global_step‚17899 _step‚1475 0loss/FRt2/assignment_loss‚0.8587109728338329 0loss/FRt1/detection_loss‚0.48276093374982154 1loss/SRqqt2/detection_loss‚0.5720383553853026 _runtime‚15342.434314373 3loss/SRqqt1/assignment_loss‚0.45224249916676484 3loss/SRqqt2/assignment_loss‚0.44450941177552467 %loss/total_loss‚7.190428279785653 epoch‚44 % _timestamp‚1.7843556837409873e+09 0loss/FRt1/assignment_loss‚1.4593659042698857 4loss/FBt1/assignment_loss‚5.5321498670145454e-05 /loss/FRt2/detection_loss‚1.2272715713430846 /loss/FBt1/detection_loss‚0.2892855174930042à ÷ÆÚSž‡7‚(:“ æ¶ìÒÙòÒ“ gpu.0.gpu‚73.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿èN­.ˆ7‚(:£ æ¶ìÒ– disk.dm-1.out‚ 1523.71484375# disk.nvme0c0n1.in‚ 25937.8046875 disk.sda3.out‚ 1523.71484375' proc.memory.availableMB‚ 369.2421875! disk.nvme0n1.in‚ 25937.8046875% disk.nvme0n1.out‚34956.8310546875 disk.rbd1.in‚ 2.1796875 proc.cpu.threads‚17$ memory_percent‚97.74632453918457 disk.sda3.in‚ 417.107421875) proc.memory.percent‚17.11559295654297 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.sda2.in‚ 82.544921875 disk.sda1.out‚0" disk.dm-0.out‚34956.8310546875 network.sent‚ 2.8759206e+07* disk./.usagePercent‚39.301943334759876 disk.sda.in‚541.0478515625 disk.sda.out‚1694.25 cpu‚26.178363792893485 network.recv‚ 4.090932e+06 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚34956.8310546875 disk.dm-0.in‚25923.76953125 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 48.2861328125 proc.memory.rssMB‚ 2804.21875$ disk./.usageGB‚585.6273574829102ÙABÇ“‰7‚(:ˆ õ¶ìÒÏšî• gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t¢mf1Š7‚(:¦ õ¶ìÒ’´ì—$ memory_percent‚97.74374961853027 cpu‚26.177908769596566) disk./.usagePercent‚39.30195357506124 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875 network.recv‚ 4.095304e+06# disk.dm-0.out‚34962.62060546875" disk.nvme0n1.in‚26079.27734375& disk.nvme0n1.out‚34962.62060546875 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625 proc.cpu.threads‚17 disk.sda.in‚541.0478515625 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 369.6640625$ disk.nvme0c0n1.in‚26079.27734375( disk.nvme0c0n1.out‚34962.62060546875 disk.dm-0.in‚ 26065.2421875 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 48.3486328125$ disk./.usageGB‚585.6275100708008 disk.dm-1.in‚ 233.44140625 disk.sda.out‚ 1694.75390625 proc.memory.rssMB‚ 2804.21875) proc.memory.percent‚17.11559295654297 disk.dm-1.out‚ 1524.21875 disk.drbd2165.out‚0 disk.sda3.out‚ 1524.21875 network.sent‚ 2.8821734e+073Tž‹7‚(:“ „·ìÒ¹øã“ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚204.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.937799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XŸ”/Œ7‚(:¤ „·ìÒÆ™– disk.rbd0.in‚ 0.16015625 cpu‚26.177465439776565 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26079.27734375) disk./.usagePercent‚39.30194973494823$ disk./.usageGB‚585.6274528503418 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875# disk.dm-0.out‚34965.15185546875 disk.sda.out‚ 1695.26953125$ memory_percent‚97.74436950683594 network.recv‚ 4.09838e+06 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1524.734375 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚ 26065.2421875 disk.sda2.in‚ 82.544921875$ proc.memory.availableMB‚369.5625 proc.cpu.threads‚17 network.sent‚ 2.8826784e+07 disk.sda3.out‚ 1524.734375 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625 proc.memory.rssMB‚ 2804.21875) proc.memory.percent‚17.11559295654297" disk.nvme0n1.in‚26079.27734375& disk.nvme0n1.out‚34965.15185546875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34965.15185546875 disk.sda.in‚541.0478515625 disk.rbd1.out‚ 48.3486328125ŠùŸ÷47/ _wandbruntime‚15381  _runtime‚15381C&G²ÉŽ7à & lr-AdamW/pg2‚1.202443981600094e-05 trainer/global_step‚17949 $ _timestamp‚1.784355722982957e+09 _runtime‚15381.676543101 _step‚1476 & lr-AdamW/pg1‚1.202443981600094e-05~þbÎ7È $ _timestamp‚1.784355722982957e+09 _runtime‚15381.676543101 _step‚1476 & lr-AdamW/pg1‚1.202443981600094e-05 & lr-AdamW/pg2‚1.202443981600094e-05 trainer/global_step‚17949Ä ñJQ47/ _wandbruntime‚15381  _runtime‚15381d%;[‘7P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142307564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70005831 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600870308235264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2732834943238968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843557229840147e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2713883025338368 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2713883008299008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18951577350710 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2727913255493632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600870326847488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19185164631040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142307566 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2727913255493632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18980878548992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142307566 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31439508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166239291 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2732834585649718 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2746893775664128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2746893775664128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620055382192128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18980537209856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70006110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38566599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2727913238454272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18980537209856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2746894134042624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18951917900600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2727913238454272 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142307564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚17949 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18980878548992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169dŒÿZ (’7†P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142307564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70005831 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600870308235264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2732834943238968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843557229840147e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2713883025338368 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2713883008299008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18951577350710 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2727913255493632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600870326847488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19185164631040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142307566 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2727913255493632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18980878548992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142307566 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31439508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166239291 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2732834585649718 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2746893775664128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2746893775664128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620055382192128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18980537209856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70006110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38566599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2727913238454272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18980537209856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2746894134042624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18951917900600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2727913238454272 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142307564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚17949 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18980878548992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169Å \‹+64“7/ _wandbruntime‚15382  _runtime‚15382sKV'”7ŠN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _runtime‚15382.29008426 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142315196 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166248343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31441256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2728060496523264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70009612 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142315196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166258334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2747042387706368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142315198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308563539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2747042387706368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620089936534016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2728060479483904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2728060496523264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18952588404432 _step‚1478 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18981549836800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚600903755606016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166248343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38568359 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573532 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142315198 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19186180928000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18981891183104 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2747042029320704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714029518970880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620089829351936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18952928954434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166258334 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18981891183104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18981549836800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169%å!=á•7N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18981891183104 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2747042029320704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714029518970880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620089829351936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18952928954434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166258334 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18981891183104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18981549836800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _runtime‚15382.29008426 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142315196 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166248343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31441256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2728060496523264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70009612 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142315196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166258334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2747042387706368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360ÂÄ 94  UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142315198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308563539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2747042387706368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚620089936534016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2728060479483904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2728060496523264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18952588404432 _step‚1478 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18981549836800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚600903755606016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166248343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38568359 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573532 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142315198 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19186180928000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983Æ º b4–7/ _wandbruntime‚15382  _runtime‚15382!麢—7Š .loss/FBt1/detection_loss‚0.379259296549373 2loss/SRqqt1/assignment_loss‚0.3861039096118475 3loss/SRqqt2/assignment_loss‚0.43668761802514405 4loss/FBt2/assignment_loss‚0.00021765501968419368 .loss/FRt1/detection_loss‚0.518017551812899 0loss/SRqqt2/detection_loss‚0.527873607760064 _step‚1479 trainer/global_step‚17949 % _timestamp‚1.7843557236004043e+09 %loss/total_loss‚6.758701583408524 epoch‚44 0loss/FRt1/assignment_loss‚1.4277251771391073 4loss/FBt1/assignment_loss‚1.7494757298922277e-05 1loss/SRqqt1/detection_loss‚0.8938114865299115 0loss/FRt2/assignment_loss‚0.6606429850963091 /loss/FRt2/detection_loss‚1.1851774403377833 0loss/FBt2/detection_loss‚0.34316736076910176 _runtime‚15382.293732268©XÐ+˜7 4loss/FBt1/assignment_loss‚1.7494757298922277e-05 1loss/SRqqt1/detection_loss‚0.8938114865299115 0loss/FRt2/assignment_loss‚0.6606429850963091 /loss/FRt2/detection_loss‚1.1851774403377833 0loss/FBt2/detection_loss‚0.34316736076910176 _runtime‚15382.293732268 .loss/FBt1/detection_loss‚0.379259296549373 2loss/SRqqt1/assignment_loss‚0.3861039096118475 3loss/SRqqt2/assignment_loss‚0.43668761802514405 4loss/FBt2/assignment_loss‚0.00021765501968419368 .loss/FRt1/detection_loss‚0.518017551812899 0loss/SRqqt2/detection_loss‚0.527873607760064 _step‚1479 trainer/global_step‚17949 % _timestamp‚1.7843557236004043e+09 %loss/total_loss‚6.758701583408524 epoch‚44 0loss/FRt1/assignment_loss‚1.4277251771391073Ç 89+’™7‚(:‡ “·ìÒ­ÿ¾“ gpu.0.gpu‚30.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žšyþ7š7‚(:¬ “·ìÒ÷©„— disk.rbd1.out‚ 48.4111328125 network.recv‚ 4.101901e+06 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 proc.memory.rssMB‚ 2804.21875 cpu‚26.177014249638088 network.sent‚ 2.8832415e+07 disk.dm-1.out‚ 1525.01171875 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0# disk.dm-0.out‚34967.93212890625$ memory_percent‚97.74744510650635* disk./.usagePercent‚39.301970983573575$ disk./.usageGB‚585.6277694702148$ disk.nvme0c0n1.in‚26079.27734375 proc.cpu.threads‚17 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1525.01171875 disk.sda.in‚541.0478515625& disk.nvme0n1.out‚34967.93212890625 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625( proc.memory.availableMB‚ 369.05859375) proc.memory.percent‚17.11559295654297" disk.nvme0n1.in‚26079.27734375 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚34967.93212890625 disk.dm-0.in‚ 26065.2421875 disk.sda.out‚ 1695.546875 disk.sda2.out‚ 170.53515625š9¾“›7‚(:ˆ ¢·ìҧ㩓 gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚225.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.0928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ºR-,œ7‚(:¡ ¢·ìÒÁÌý• disk.dm-1.in‚ 233.44140625% disk.nvme0n1.out‚34971.0615234375 network.recv‚ 4.10647e+06" disk.nvme0n1.in‚26079.27734375 proc.memory.rssMB‚ 2804.21875 disk.sda.out‚ 1696.04296875 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1525.5078125 disk.drbd2165.out‚0 cpu‚26.176570501582884$ disk.nvme0c0n1.in‚26079.27734375$ memory_percent‚97.74627685546875" proc.memory.availableMB‚369.25) disk./.usagePercent‚39.30196432737768" disk.dm-0.out‚34971.0615234375 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1525.5078125 disk.sda1.out‚0$ disk./.usageGB‚585.6276702880859 disk.rbd1.in‚ 2.1796875 disk.drbd2165.in‚ 2.25390625) proc.memory.percent‚17.11559295654297 disk.dm-0.in‚ 26065.2421875 disk.rbd0.in‚ 0.16015625 proc.cpu.threads‚17' disk.nvme0c0n1.out‚34971.0615234375 network.sent‚ 2.8896405e+07 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 48.4111328125¨,ùÈ“7‚(:ˆ ±·ìÒ£“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¥¯|4ž7‚(:© ±·ìÒב͕ network.sent‚ 2.8901555e+07 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 48.4736328125 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.dm-0.in‚ 26169.9609375" disk.nvme0n1.in‚26183.99609375 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625 network.recv‚ 4.110068e+06 disk.dm-1.out‚ 1525.80859375( disk.nvme0c0n1.out‚34976.99560546875 disk.sda2.out‚ 170.53515625( proc.memory.availableMB‚ 369.19921875$ disk./.usageGB‚585.6277389526367 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚26183.99609375# disk.dm-0.out‚34976.99560546875& disk.nvme0n1.out‚34976.99560546875 disk.sda3.out‚ 1525.80859375 disk.dm-1.in‚ 233.44140625 cpu‚26.176092935645944 proc.cpu.threads‚17 disk.sda.out‚ 1696.34375 disk.sda1.out‚0 proc.memory.rssMB‚ 2804.21875) proc.memory.percent‚17.11559295654297( disk./.usagePercent‚39.3019689355133 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.746586799621587m³4Ÿ7/ _wandbruntime‚15421  _runtime‚15421D߃¼Ë 7Å % _timestamp‚1.7843557630063229e+09 _runtime‚15421.69983447 _step‚1480 ' lr-AdamW/pg1‚1.1450288242276917e-05 ' lr-AdamW/pg2‚1.1450288242276917e-05 trainer/global_step‚17999F$†òС7Ê ' lr-AdamW/pg2‚1.1450288242276917e-05 trainer/global_step‚17999 % _timestamp‚1.7843557630063229e+09 _runtime‚15421.69983447 _step‚1480 ' lr-AdamW/pg1‚1.1450288242276917e-05È º >4¢7/ _wandbruntime‚15421  _runtime‚15421çÄ (£7€P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31526758 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166703720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142689166 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1481 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142689166 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19031574050816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602541764773888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38669502 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2735275289935872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19031232711680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843557630073454e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2735275306975232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166693743 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2754306522647552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2740210735730226 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19236226040320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2754306881026048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19031232711680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309392886 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166693743 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2735275289935872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19031574050816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31526761 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2754306881026048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70196260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2721208541892608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19236306228736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142689164 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621777972201984 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19002193837618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142689164 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38669760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166703720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602541746161664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚15421.701055983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309382907 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2735275306975232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621778071002624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70196521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088Z¦7 (¤7…P KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70196521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31526758 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166703720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142689166 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1481 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142689166 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19031574050816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602541764773888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38669502 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2735275289935872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19031232711680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843557630073454e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2735275306975232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166693743 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2754306522647552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2740210735730226 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19236226040320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2754306881026048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19031232711680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309392886 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166693743 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2735275289935872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19031574050816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31526761 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2754306881026048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70196260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2721208541892608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19236306228736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142689164 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621777972201984 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19002193837618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142689164 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38669760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166703720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602541746161664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚15421.701055983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309382907 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2735275306975232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621778071002624É Ù#(Ç4¥7/ _wandbruntime‚15422  _runtime‚15422 ~´K^ ¦7ŠN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621812429363712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2721355052564480 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2754455134689792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309409584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70200326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38671820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843557636227689e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166712786 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2735422530965504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31528506 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19003204891340 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166702795 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2735422548004864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1482 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38671562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19032245338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2754455134689792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19003545441342 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19032586684928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166712786 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142696798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166702795 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602575174920192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19032586684928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70200065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19237334624768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621812528157184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38671562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19032245338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2754455134689792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19003545441342 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19032586684928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166712786 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142696798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166702795 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602575174920192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19032586684928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70200065 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19237334624768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621812528157184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309409584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70200326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38671820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843557636227689e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166712786 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2735422530965504 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31528506 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19003204891340 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166702795 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2735422548004864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1482 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23891869187355042 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚44 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23750638961791992 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6849631071090698 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2834002375602722 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9431183934211731 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.693467915058136 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09121575206518173 +jet/accuracy_3_of_4‚0.04078979045152664 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7660830616950989 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3808146119117737 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3399360179901123 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _step‚1484 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7741742134094238 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6847072243690491 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6730299592018127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24410729110240936 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.716439962387085 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28400325775146484 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.79609614610672 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047015540301799774 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23391211032867432 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3666521906852722 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7867524027824402 particle/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.785014808177948 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 'particle/f_score‚0.6055139303207397 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7794028520584106 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8724486827850342 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24861539900302887 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7118048667907715 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714718103408813 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7269617915153503 *jet/accuracy_1_of_3‚0.9288378953933716 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5642223954200745 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3169514834880829 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25005286931991577 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6946977376937866 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036708783358335495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_2_of_3‚0.5115707516670227 particle/accuracy_2_of_2‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09954337030649185 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2840701639652252 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8355631232261658 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_3‚0.03840682655572891 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16724076867103577 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21335788071155548 *jet/accuracy_1_of_4‚0.9806751012802124 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025844940915703773 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025699149817228317 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5043005347251892 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7292292714118958 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7100115418434143 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943199872970581 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8306500315666199 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03679104894399643 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1364985704421997 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7752801179885864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27625641226768494 +particle/sensitivity‚0.5951111316680908 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6719049215316772 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3796260952949524 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7677456140518188 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510037124156952 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29438507556915283 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2756662368774414 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6336256861686707 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5320982336997986 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803701996803284 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29435473680496216 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6433060169219971 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03658539056777954 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2215658575296402 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8724486827850342 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.724464831785364 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 (particle/accuracy‚0.8334843516349792 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 jet/accuracy_1_of_5‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226717472076416 5 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.62603360414505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6174098253250122 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.642540454864502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7321429252624512 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21396857500076294 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.747WºòÕ 1622824668884 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.642540454864502 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5480099320411682 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7294678688049316 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7946341633796692 +jet/accuracy_2_of_2‚0.12676827609539032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 trainer/global_step‚17999 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15056312084197998 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3305456042289734 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6540195345878601 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28400325775146484 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007004737854004 +particle/specificity‚0.8986762762069702 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02588457427918911 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02561570331454277 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05257368087768555 particle/accuracy_2_of_3‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6586702466011047 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15293154120445251 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.293912798166275 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5706431269645691 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6323628425598145 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 *jet/accuracy_1_of_1‚0.3481091260910034 _runtime‚15439.480791228 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23953962326049805 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.792467474937439 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23698790371418 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.657219409942627 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10217443853616714 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7276082634925842 particle/accuracy_3_of_3‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.78702425956726076¾aù€=¯7úz ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10217443853616714 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7276082634925842 particle/accuracy_3_of_3‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7870242595672607 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12995494902133942 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6612937450408936 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1458718478679657 particle/accuracy_1_of_2‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7671706080436707 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843557807865038e+09 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19254346191883087 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23891869187355042 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚44 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23750638961791992 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6849631071090698 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2834002375602722 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9431183934211731 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.693467915058136 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09121575206518173 +jet/accuracy_3_of_4‚0.04078979045152664 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7660830616950989 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3808146119117737 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3399360179901123 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _step‚1484 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7741742134094238 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6847072243690491 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6730299592018127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24410729110240936 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.716439962387085 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28400325775146484 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.79609614610672 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047015540301799774 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23391211032867432 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3666521906852722 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7867524027824402 particle/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.785014808177948 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 'particle/f_score‚0.6055139303207397 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7794028520584106 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8724486827850342 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24861539900302887 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7118048667907715 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714718103408813 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7269617915153503 *jet/accuracy_1_of_3‚0.9288378953933716 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5642223954200745 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3169514834880829 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25005286931991577 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6946977376937866 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036708783358335495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_2_of_3‚0.5115707516670227 particle/accuracy_2_of_2‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09954337030649185 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2840701639652252 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8355631232261658 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_3‚0.03840682655572891 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16724076867103577 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21335788071155548 *jet/accuracy_1_of_4‚0.9806751012802124 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025844940915703773 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025699149817228317 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5043005347251892 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7292292714118958 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7100115418434143 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943199872970581 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8306500315666199 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03679104894399643 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1364985704421997 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7752801179885864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27625641226768494 +particle/sensitivity‚0.5951111316680908 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6719049215316772 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3796260952949524 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7677456140518188 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510037124156952 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29438507556915283 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2756662368774414 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6336256861686707 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5320982336997986 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803701996803284 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29435473680496216 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6433060169219971 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03658539056777954 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2215658575296402 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8724486827850342 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.724464831785364 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 (particle/accuracy‚0.8334843516349792 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 jet/accuracy_1_of_5‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226717472076416 5 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.62603360414505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6174098253250122 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.642540454864502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7321429252624512 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21396857500076294 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7471622824668884 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.642540454864502 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5480099320411682 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7294678688049316 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7946341633796692 +jet/accuracy_2_of_2‚0.12676827609539032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 trainer/global_step‚17999 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15056312084197998 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3305456042289734 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6540195345878601 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28400325775146484 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007004737854004 +particle/specificity‚0.8986762762069702 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02588457427918911 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02561570331454277 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05257368087768555 particle/accuracy_2_of_3‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6586702466011047 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15293154120445251 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.293912798166275 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5706431269645691 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6323628425598145 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 *jet/accuracy_1_of_1‚0.3481091260910034 _runtime‚15439.480791228 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23953962326049805 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.792467474937439 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23698790371418 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.657219409942627Ì s`.“°7‚(:ˆ Ï·ìÒ­‰¦“ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßPû-±7‚(:¢ Ï·ìÒ†ìÙ• disk.sda.out‚ 1697.00390625 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1526.46875 disk.rbd1.in‚ 2.1796875) disk./.usagePercent‚39.30193130240577 disk.sda.in‚541.0478515625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2798.89453125 disk.dm-1.out‚ 1526.46875$ disk.nvme0c0n1.in‚26183.99609375! disk.dm-0.out‚34986.166015625$ disk.nvme0n1.out‚34986.166015625 disk.sda3.in‚ 417.107421875 disk.sda1.out‚0 cpu‚26.1796275724018 network.recv‚ 4.116771e+06 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚34986.166015625* proc.memory.percent‚17.083096504211426" disk.nvme0n1.in‚26183.99609375 network.sent‚ 2.8968765e+07 disk.dm-0.in‚ 26169.9609375 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 48.5361328125 disk.sda1.in‚ 8.759765625$ memory_percent‚97.68643379211426' proc.memory.availableMB‚ 379.0546875$ disk./.usageGB‚585.6271781921387ù쓲7‚(:ˆ Þ·ìÒòܬ“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚244.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JƒA &³7‚(:› Þ·ìÒÏ÷Ó•( proc.memory.availableMB‚ 379.33984375 disk.dm-1.in‚ 233.44140625$ memory_percent‚97.68469333648682* disk./.usagePercent‚39.301938982631796" disk.nvme0n1.in‚26183.99609375 disk.sda3.in‚ 417.107421875 disk.dm-0.in‚ 26169.9609375* proc.memory.percent‚17.083096504211426 cpu‚26.179179534708368 disk.sda.out‚ 1697.31640625 disk.sda2.in‚ 82.544921875 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1526.78125 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1526.78125 network.sent‚ 2.902599e+07# proc.memory.rssMB‚ 2798.89453125 disk.sda2.out‚ 170.53515625 disk.drbd2165.out‚0! disk.nvme0n1.out‚ 34989.171875 disk.rbd1.out‚ 48.5361328125 disk.drbd2165.in‚ 2.25390625 disk.dm-0.out‚ 34989.171875 proc.cpu.threads‚17 network.recv‚ 4.12074e+06$ disk./.usageGB‚585.6272926330566 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26183.99609375# disk.nvme0c0n1.out‚ 34989.171875)€"ç4´7/ _wandbruntime‚15479  _runtime‚15479¬ù)˵7Å ' lr-AdamW/pg1‚1.0889822924364737e-05 ' lr-AdamW/pg2‚1.0889822924364737e-05 trainer/global_step‚18049 $ _timestamp‚1.784355820314358e+09 _runtime‚15479.007917868 _step‚1485£éж7Ê _step‚1485 ' lr-AdamW/pg1‚1.0889822924364737e-05 ' lr-AdamW/pg2‚1.0889822924364737e-05 trainer/global_step‚18049 $ _timestamp‚1.784355820314358e+09 _runtime‚15479.007917868Í Ý’b4·7/ _wandbruntime‚15479  _runtime‚15479¡¢íSâ&¸7€P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19070530049755 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19100002758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143191432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2763666927865344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31616531 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2763666569486848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167218517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31616528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38804698 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38804948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143191430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19100002758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167228494 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19099661419008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19099661419008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2763666569486848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70421226 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843558203154492e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2744566908067840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167228494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚623534804882432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2744566925107200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19070870599645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2749513626339293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2763666927865344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2744566925107200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚18049 _step‚1486 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143191432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚604229096187904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚15479.009290059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604229114800128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310409947 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143191430 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚623534903683072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2730442755739648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚15479.009290059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604229114800128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310409947 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143191430 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚623534903683072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2730442755739648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31616531 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2763666569486848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167218517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31616528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38804698 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38804948 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143191430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19100002758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167228494 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19099661419008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19099661419008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2763666569486848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70421226 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843558203154492e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2744566908067840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167228494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚623534804882432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2744566925107200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19070870599645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2749513626339293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2763666927865344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Î ] uÑ4º7/ _wandbruntime‚15479  _runtime‚15479é 2'»7ŠN KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143199064 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167227569 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2763814823143424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19306742423040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70425277 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19071881653479 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843558209295814e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310426631 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19101015392256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143199062 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2749660790475637 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623569366162944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15479.623507937 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310436624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38806741 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19306822604288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18049 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310426631 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2749661148064999 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2744714149097472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38807001 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604262524946432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2763815181529088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2744714149097472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19101015392256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310436624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31618273 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚1487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604262543558656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2730589249372160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2730589266411520 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70425014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2763815181529088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2744714166136832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19071541103477 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19100674045952$…ÚÉ'¼7N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2730589266411520 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70425014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2763815181529088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2744714166136832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19071541103477 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19100674045952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143199064 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167227569 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2763814823143424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19306742423040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70425277 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19071881653479 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843558209295814e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310426631 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19101015392256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143199062 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2749660790475637 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623569366162944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15479.623507937 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310436624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38806741 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19306822604288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18049 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310426631 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2749661148064999 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2744714149097472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38807001 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604262524946432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2763815181529088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2744714149097472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19101015392256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310436624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31618273 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚1487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604262543558656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2730589249372160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576Ï gð^Ê4½7/ _wandbruntime‚15479  _runtime‚15479U)hy¾7ˆ 0loss/FBt1/detection_loss‚0.24829203944318579 /loss/FRt2/assignment_loss‚0.865497272907537 2loss/SRqqt1/assignment_loss‚0.4014566734311115 3loss/FBt1/assignment_loss‚0.0010290697981134083 4loss/FBt2/assignment_loss‚0.00010527331166879963 _step‚1488 0loss/FRt1/assignment_loss‚1.4564216847675455 .loss/FRt2/detection_loss‚1.157473262140984 1loss/SRqqt2/detection_loss‚0.5591926125416238 %loss/total_loss‚6.841386323780036 epoch‚45 _runtime‚15479.625868814 1loss/SRqqt2/assignment_loss‚0.399781090093957 /loss/FBt2/detection_loss‚0.3271768753417836 trainer/global_step‚18049 % _timestamp‚1.7843558209325747e+09 /loss/FRt1/detection_loss‚0.5607151384908371 1loss/SRqqt1/detection_loss‚0.8642453315116885Ij—T¿7 1loss/SRqqt2/assignment_loss‚0.399781090093957 /loss/FBt2/detection_loss‚0.3271768753417836 trainer/global_step‚18049 % _timestamp‚1.7843558209325747e+09 /loss/FRt1/detection_loss‚0.5607151384908371 1loss/SRqqt1/detection_loss‚0.8642453315116885 0loss/FBt1/detection_loss‚0.24829203944318579 /loss/FRt2/assignment_loss‚0.865497272907537 2loss/SRqqt1/assignment_loss‚0.4014566734311115 3loss/FBt1/assignment_loss‚0.0010290697981134083 4loss/FBt2/assignment_loss‚0.00010527331166879963 _step‚1488 0loss/FRt1/assignment_loss‚1.4564216847675455 .loss/FRt2/detection_loss‚1.157473262140984 1loss/SRqqt2/detection_loss‚0.5591926125416238 %loss/total_loss‚6.841386323780036 epoch‚45 _runtime‚15479.625868814Ð »òÉ¡À7‚(:’ í·ìÒá gpu.0.gpu‚50.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cž5@Á7‚(:­ í·ìÒÖšã•) disk./.usagePercent‚39.3019305343ëÿòËø8316$ disk./.usageGB‚585.6271667480469 disk.dm-0.in‚ 26273.8515625# disk.dm-0.out‚34994.92431640625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1527.07421875 disk.drbd2165.out‚0$ memory_percent‚97.68495559692383 cpu‚26.178731769870744 network.sent‚ 2.9031069e+07 disk.sda3.out‚ 1527.07421875* proc.memory.percent‚17.083096504211426 disk.sda.out‚ 1697.609375 disk.rbd1.out‚ 48.5986328125 disk.drbd2165.in‚ 2.25390625 disk.sda1.out‚0# proc.memory.rssMB‚ 2798.89453125$ disk.nvme0c0n1.in‚26287.88671875 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚34994.92431640625 disk.rbd0.in‚ 0.16015625 proc.cpu.threads‚17 network.recv‚ 4.124187e+06( disk.nvme0c0n1.out‚34994.92431640625 disk.sda.in‚541.0478515625" disk.nvme0n1.in‚26287.88671875 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.sda2.in‚ 82.544921875& proc.memory.availableMB‚ 379.296875O`‘Â7‚(:† ü·ìÒ”⯓ gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÍ8Ã7‚(:­ ü·ìÒËÁ¶– disk.drbd2165.out‚0 network.sent‚ 2.9094117e+07" disk.nvme0n1.in‚26287.88671875 disk.rbd1.out‚ 48.5986328125 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 network.recv‚ 4.128534e+06 disk.sda.out‚ 1697.93359375 disk.dm-1.out‚ 1527.3984375$ disk.nvme0c0n1.in‚26287.88671875 disk.dm-0.in‚ 26273.8515625 disk.sda3.out‚ 1527.3984375# proc.memory.rssMB‚ 2798.89453125$ disk./.usageGB‚585.6198921203613 disk.sda.in‚541.0478515625& disk.nvme0n1.out‚34998.91650390625( disk.nvme0c0n1.out‚34998.91650390625# disk.dm-0.out‚34998.91650390625* proc.memory.percent‚17.083096504211426) disk./.usagePercent‚39.30144232801537 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 cpu‚26.17826696060219 disk.sda2.in‚ 82.544921875 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17$ memory_percent‚97.68691062927246' proc.memory.availableMB‚ 378.9765625 disk.sda2.out‚ 170.53515625 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625BZ”žÄ7‚(:“ ‹¸ìÒ㬮“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üe’X,Å7‚(:¡ ‹¸ìÒΣՖ disk.sda.in‚541.0478515625 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0 proc.cpu.threads‚17$ memory_percent‚97.68850803375244 disk.sda.out‚1698.375 network.sent‚ 2.9099163e+07 disk.dm-1.out‚ 1527.83984375 disk.dm-0.in‚ 26273.8515625 disk.sda3.out‚ 1527.83984375# proc.memory.rssMB‚ 2798.89453125 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26287.88671875 disk.sda2.in‚ 82.544921875$ disk./.usageGB‚585.6199150085449% disk.nvme0c0n1.out‚35005.07421875# disk.nvme0n1.out‚35005.07421875 disk.sda1.in‚ 8.759765625" disk.nvme0n1.in‚26287.88671875 disk.sda1.out‚0) disk./.usagePercent‚39.30144386406057 disk.dm-0.out‚35005.07421875 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 48.6611328125 disk.rbd0.in‚ 0.16015625* proc.memory.percent‚17.083096504211426( proc.memory.availableMB‚ 378.71484375 network.recv‚ 4.132105e+06 disk.sda3.in‚ 417.107421875 cpu‚26.1778204679942Jø4Æ7/ _wandbruntime‚15519  _runtime‚15519°*Š0ÉÇ7à _step‚1489 & lr-AdamW/pg1‚1.034307985990458e-05 & lr-AdamW/pg2‚1.034307985990458e-05 trainer/global_step‚18099 $ _timestamp‚1.784355860394552e+09 _runtime‚15519.088117954ÌЗÎÈ7È $ _timestamp‚1.784355860394552e+09 _runtime‚15519.088117954 _step‚1489 & lr-AdamW/pg1‚1.034307985990458e-05 & lr-AdamW/pg2‚1.034307985990458e-05 trainer/global_step‚18099Ñ ­J‘ƒ4É7/ _wandbruntime‚15519  _runtime‚15519v·ªk(Ê7P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38908562 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625257903732224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605900534114304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19357268720128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1490 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2771079316470272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311255978 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38908835 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31703778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15519.089408871 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167672969 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2771079674848768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19150698259968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2751928959549440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605900552726528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2737768272293888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19150356920832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311245999 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19121487086553 % _timestamp‚1.7843558603957038e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1490 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2771079316470272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311255978 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38908835 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31703778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15519.089408871 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167672969 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2771079674848768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19150698259968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2751928959549440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605900552726528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2737768272293888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19150356920832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311245999 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19121487086553 % _timestamp‚1.7843558603957038e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311272676 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311272676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38910912 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2752076200579072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2737914800005120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311262683 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19151710894080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19151369547776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70616438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2771227570126848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625292261069824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38910654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625292359863296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2757036940556145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143580664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311262683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19358378378240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19358298196992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19122498140387 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167692012 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2737914782965760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143580662 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚1491 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚18099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19151710894080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143580664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2752076200579072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605933981485056 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2752076217618432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843558610115902e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70616177†³ª'Î7N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167682021 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19122157590385 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2771227928512512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167692012 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143580662 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70616177 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311272676 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311272676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38910912 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2752076200579072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2737914800005120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311262683 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19151710894080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19151369547776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70616438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2771227570126848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625292261069824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38910654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625292359863296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2757036940556145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143580664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311262683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19358378378240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19358298196992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19122498140387 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167692012 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2737914782965760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143580662 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚1491 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚18099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19151710894080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143580664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2752076200579072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605933981485056 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2752076217618432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843558610115902e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168127421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31791028 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19172103573461 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312092030 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168137398 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168137398 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70803714 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2778492063453696 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2759291011031040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312082051 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143954630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19201052422656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19201052422656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2778492421832192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19408847262208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764265926500309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70803443 trainer/global_step‚18149 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2759291028070400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143954630 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39012683 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31791031Os ˆ Ý7„P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2759291028070400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143954630 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39012683 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31791031 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1494 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2745093822926848 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2745093805887488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚15558.97653418 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚607571972040704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843559002828107e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚626980745405952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168127421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31791028 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19172103573461 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_spzQ‚lit.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312092030 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168137398 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168137398 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70803714 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2778492063453696 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2759291011031040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312082051 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143954630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19201052422656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19201052422656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2778492421832192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19408847262208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764265926500309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70803443 trainer/global_step‚18149 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚627015196432384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2745240316559360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607605400799232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚18149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19173114627295 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168146464 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168136473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2745240333598720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2778640675495936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312098735 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70807536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19172774077293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70807279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2764413090636653 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312108728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚15559.58959053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31792773 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19202065049600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143962262 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2778640675495936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19202065049600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168136473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1495 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19202406395904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2759438269100032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143962262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2778640317110272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚627015297323008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2759438252060672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19409795633152Žµú'à7N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312098735 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 # _timestamp‚1.78435590089581e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312108728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚15559.58959053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31792773 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19202065049600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143962262 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2778640675495936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19202065049600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168136473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1495 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19202406395904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2759438269100032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143962262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2778640317110272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚627015297323008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2759438252060672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19409795633152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚627015196432384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2745240316559360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607605400799232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚18149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19173114627295 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168146464 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168136473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2745240333598720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2778640675495936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312098735 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70807536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19172774077293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70807279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2764413090636653 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2785904810437120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19251747924480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144336230 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2766653079552000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.¾C<Å(%freed‚628703407729664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2771642076580817 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1498 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39116349 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19252089263616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19251747924480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚18199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312918103 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2766653062512640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168581873 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19460098922496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19222720060369 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2785905168815616 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2766653062512640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609243409967104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2771641718991567 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2752419339481088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚15599.027966878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144336232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628703527501824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609243428579328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2766653079552000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 $ _timestamp‚1.784355940334277e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2785904810437120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168591850 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19222379510479 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2785905168815616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70994630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19252089263616@ø] (í7†P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19460098922496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19222720060369 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2785905168815616 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2766653062512640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609243409967104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2771641718991567 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2752419339481088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚15599.027966878 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144336232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628703527501824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609243428579328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2766653079552000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 $ _timestamp‚1.784355940334277e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2785904810437120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168591850 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19222379510479 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2785905168815616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70994630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19252089263616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168581873 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2785904810437120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19251747924480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144336230 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2766653079552000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628703407729664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2771642076580817 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1498 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39116349 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19252089263616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19251747924480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚18199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312918103 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2766653062512640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168581873Ú i5ÈÂ4î7/ _wandbruntime‚15599  _runtime‚15599¨çV0'ï7‹N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168600916 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609276838725632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19461111475712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2786053422479360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19253101897728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168600916 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168590925 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70998088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1499 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144343862 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144343862 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628737968813568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2766800320581632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168590925 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19252760551424 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19223390564201 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144343864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312934787 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312934787 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19461020808704 _runtime‚15599.641620199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628737859534336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2752565867192320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312944780 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70998349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2766800303542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19252760551424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144343864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 % _timestamp‚1.7843559409479165e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31880023 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31880026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2771789598306523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39118065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚609276857337856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39118323 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 m¼id ð7N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19253101897728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2771789240717161 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19223731114203 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2766800320581632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168590925 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19252760551424 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19223390564201 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144343864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312934787 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/ƒ¦Ác²allocation.all.freed‚ 312934787 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19461020808704 _runtime‚15599.641620199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628737859534336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2752565867192320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312944780 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70998349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2766800303542272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19252760551424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144343864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 % _timestamp‚1.7843559409479165e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31880023 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31880026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2771789598306523 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39118065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚609276857337856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39118323 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168600916 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609276838725632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19461111475712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2786053422479360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19253101897728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168600916 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168590925 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70998088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1499 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144343862 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144343862 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628737968813568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EÔÏü7É & lr-AdamW/pg2‚8.785530929840006e-06 trainer/global_step‚18249 % _timestamp‚1.7843559801422784e+09 _runtime‚15638.835814367 _step‚1501 & lr-AdamW/pg1‚8.785530929840006e-06Ý Œ£4ý7/ _wandbruntime‚15638  _runtime‚15638Æé|(þ7€P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19302784765440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚18249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31965528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313754155 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169036325 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774015131033600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2793317557420544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2793317557420544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19302443426304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169036325 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71185379 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784355980143332e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313764134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39219848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31965531 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144717832 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2774015131033600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630426278606848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚15638.837023641 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169046302 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144717832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19302443426304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2793317915799040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144717832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19302443426304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2793317915799040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774015131033600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2793317557420544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2793317557420544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19302443426304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169036325 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71185379 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784355980143332e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313764134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39219848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Þ 2Š#A4€8/ _wandbruntime‚15639  _runtime‚15639º‹òà'8‹N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169045377 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610948276652032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169045377 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169055368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144725462 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19303797399552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313780832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2774162372063232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610948295264256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1503 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2774162355023872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19512434984448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169055368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39221640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19512346414592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2759891400785920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144725464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2793466169462784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774162355023872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630460730248704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313770839 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71188913 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 _runtime‚15639.451275563 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71189167 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2779165748387031 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19274347601111 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19303456053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313770839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19274007051109 trainer/global_step‚18249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144725464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2793466169462784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2793465811077120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313780832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31967276 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19274007051109 trainer/global_step‚18249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144725464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2793466169462784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2793465811077120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313780832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31967276 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169055368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39221640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19512346414592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2759891400785920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144725464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2793466169462784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774162355023872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630460730248704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313770839 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71188913 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 _runtime‚15639.451275563 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71189167 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2779165748387031 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19274347601111 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19303456053248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313770839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2800730304403968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2781377182515200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169490777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19562604311552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145099430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632148799457280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2767070406668288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2767070423707648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2781377165475840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314600186 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDev…_{iceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32052781 % _timestamp‚1.7843560201778212e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32052778 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚15678.871548886 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19353138928128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19323612484295 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145099430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19562513637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19353138928128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632148908743680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2800730304403968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169500754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2800730662782464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2781377165475840 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2786394376741833 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19323953034185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2781377182515200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39323261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚612586304432128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚612586285819904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1506 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101rþYÙ (‘8†P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2781377165475840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314600186 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32052781 % _timestamp‚1.7843560201778212e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32052778 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚15678.871548886 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19353138928128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19323612484295 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145099430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19562513637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19353138928128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632148908743680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2800730304403968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169500754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2800730662782464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2781377165475840 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2786394376741833 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19323953034185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2781377182515200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39323261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚612586304432128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚612586285819904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1506 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2786394019152583 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145099432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19353480267264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145099432 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169500754 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2800730304403968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2781377182515200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169490777 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19562604311552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145099430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632148799457280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2767070406668288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2767070423707648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0â ª<#4’8/ _wandbruntime‚15679  _runtime‚15679ô"»~'“8‹N WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2781524423544832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2767216917340160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314606891 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚632183360918016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2781524423544832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2800878916446208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2800878558060544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145107062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19563627727360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2786541540878177 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2767216934379520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2781524406505472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314606891 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71379858 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19354492901376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32054526 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1507 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145107064 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2800878916446208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632183253735936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314616884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚18299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71379600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2781524423544832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2800878916446208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2800878558060544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145107062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19563627727360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2786541540878177 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2767216934379520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2781524406505472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314606891 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71379858 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19354492901376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32054526 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1507 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145107064 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2800878916446208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632183253735936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314616884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚18299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71379600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ã ì=84•8/ _wandbruntime‚15679  _runtime‚15679†á%¨–8Š epoch‚45 0loss/FRt1/assignment_loss‚1.4028582040199453 2loss/SRqqt1/assignment_loss‚0.4842051715856915 /loss/FRt2/detection_loss‚1.2270080512531623 /loss/FBt1/detection_loss‚0.3758763333387143 _step‚1508 0loss/FRt2/assignment_loss‚0.7708365236680339 3loss/SRqqt2/assignment_loss‚0.42445346338435685 2loss/FBt2/assignment_loss‚0.004533165344343134 /loss/FRt1/detection_loss‚0.6034296380630776 1loss/SRqqt2/detection_loss‚0.5714081487535276 % _timestamp‚1.7843560207949498e+09 4loss/FBt1/assignment_loss‚1.8060486422557225e-05 1loss/SRqqt1/detection_loss‚0.9393872605326774 /loss/FBt2/detection_loss‚0.4717129100947234 %loss/total_loss‚7.275726930524675 trainer/global_step‚18299 _runtime‚15679.488236529z1³)—8 0loss/FRt2/assignment_loss‚0.7708365236680339 3loss/SRqqt2/assignment_loss‚0.42445346338435685 2loss/FBt2/assignment_loss‚0.004533165344343134 /loss/FRt1/detection_loss‚0.6034296380630776 1loss/SRqqt2/detection_loss‚0.5714081487535276 % _timestamp‚1.7843560207949498e+09 4loss/FBt1/assignment_loss‚1.8060486422557225e-05 1loss/SRqqt1/detection_loss‚0.9393872605326774 /loss/FBt2/detection_loss‚0.4717129100947234 %loss/total_loss‚7.275726930524675 trainer/global_step‚18299 _runtime‚15679.488236529 epoch‚45 0loss/FRt1/assignment_loss‚1.4028582040199453 2loss/SRqqt1/assignment_loss‚0.4842051715856915 /loss/FRt2/detection_loss‚1.2270080512531623 /loss/FBt1/detection_loss‚0.3758763333387143 _step‚1508ä {”ú“˜8‚(:ˆ ¿¹ìÒª§÷• gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚245.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 mè˜3™8‚(:¨ ¿¹ìҳ network.sent‚ 2.9392471e+07 disk.rbd0.in‚ 0.16015625# proc.memory.rssMB‚ 2798.89453125 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625$ disk./.usageGB‚585.6280097961426 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625$ memory_percent‚97.68972396850586 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625$ disk.nvme0c0n1.in‚26621.32421875 disk.sda.out‚ 1703.4921875 disk.sda3.out‚ 1532.95703125 disk.rbd1.out‚ 49.1611328125 disk.sda1.in‚ 8.759765625" disk.nvme0n1.in‚26621.32421875 network.recv‚ 4.176456e+06 disk.dm-1.out‚ 1532.95703125 disk.dm-0.in‚ 26607.2890625 disk.sda2.out‚ 170.53515625$ disk.nvme0n1.out‚35063.310546875& proc.memory.availableMB‚ 378.515625* proc.memory.percent‚17.083096504211426 cpu‚26.172490110783343) disk./.usagePercent‚39.30198711204823& disk.nvme0c0n1.out‚35063.310546875! disk.dm-0.out‚35063.310546875 proc.cpu.threads‚17 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875&µµžš8‚(:“ ιìÒ¾ó” gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.616' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.923199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<úÌ9›8‚(:® ιìÒ—ž– disk.drbd2165.in‚ 2.25390625 disk.sda3.in‚ 417.107421875$ memory_percent‚97.69032001495361" disk.nvme0n1.in‚26621.32421875 network.recv‚ 4.180715e+06) disk./.usagePercent‚39.30198788007083 disk.rbd1.out‚ 49.1611328125 disk.sda.out‚ 1703.828125 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚35066.25244140625 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.sda3.out‚ 1533.29296875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35066.25244140625 disk.dm-0.in‚ 26607.2890625# disk.dm-0.out‚35066.25244140625 disk.sda.in‚541.0478515625$ disk./.usageGB‚585.6280212402344 disk.rbd0.in‚ 0.16015625 disk.sda2.in‚ 82.544921875 cpu‚26.172074736906733 disk.rbd1.in‚ 2.1796875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2798.89453125* proc.memory.percent‚17.083096504211426 proc.cpu.threads‚17 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1533.29296875 network.sent‚ 2.945504e+07$ disk.nvme0c0n1.in‚26621.32421875( proc.memory.availableMB‚ 378.417968753R†C4œ8/ _wandbruntime‚15718  _runtime‚15718õüFxÊ8Ä & lr-AdamW/pg1‚7.816396437241126e-06 & lr-AdamW/pg2‚7.816396437241126e-06 trainer/global_step‚18349 % _timestamp‚1.7843560600287902e+09 _runtime‚15718.722355408 _step‚1509®, gÏž8É _runtime‚15718.722355408 _step‚1509 & lr-AdamW/pg1‚7.816396437241126e-06 & lr-AdamW/pg2‚7.816396437241126e-06 trainer/global_step‚18349 % _timestamp‚1.7843560600287902e+09å Ç@´Æ4Ÿ8/ _wandbruntime‚15718  _runtime‚15718]×U+( 8P TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788739233996800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169955206 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614257742358528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169945229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39426369 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71566397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2793770169233091 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145481032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19403834429952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15718.723583347 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19613775625728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2774395957301248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19374569521093 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145481030 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145481030 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19404175769088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614257723746304 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169945229 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2808143051387392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32140031 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2774395940261888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2808143409765888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843560600298696e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315426259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2808143051387392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633871402406400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19374228971203 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169955206 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2788739216957440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19404175769088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2694j^ (¡8†P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169945229 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2808143051387392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32140031 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2774395940261888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2808143409765888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7843560600298696e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315426259 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2808143051387392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633871402406400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19374228971203 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169955206 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2788739216957440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19404175769088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788739233996800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169955206 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚614257742358528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169945229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39426369 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71566397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2793770169233091 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145481032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19403834429952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15718.723583347 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19613775625728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2774395957301248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19374569521093 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145481030 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145481030 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19404175769088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614257723746304æ BC£4¢8/ _wandbruntime‚15719  _runtime‚15719b/i±M£8‹N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2808291663429632 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19404847056896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2808291305043968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633905856406528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19614703901696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2788886475026432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2774542450933760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19404847056896 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71570202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71570477 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315452936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145488662 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169964272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19375580574927 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843560608314915e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@Devi$×ýÄceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788886457987072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2788886457987072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19405188403200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19614790374400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315442943 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚633905961491456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18349 _runtime‚15719.525398153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2808291663429632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614291171117056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169964272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32141776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145488662 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315442943 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614291152504832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169954281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32141773 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2793917690958685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Jç Ÿ'¤8N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315442943 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚633905961491456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18349 _runtime‚15719.525398153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2808291663429632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614291171117056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169964272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32141776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145488662 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315442943 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚614291152504832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169954281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32141773 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2793917690958685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2808291663429632 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19404847056896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2808291305043968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633905856406528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19614703901696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2788886475026432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2774542450933760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19404847056896 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71570202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71570477 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315452936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145488662 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169964272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19375580574927 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843560608314915e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788886457987072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2788886457987072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19405188403200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19614790374400ç x“;¸4¥8/ _wandbruntime‚15719  _runtime‚15719‹¦Î» ¦8… 3loss/FBt1/assignment_loss‚3.228578565976174e-07 1loss/SRqqt2/detection_loss‚0.5665353589618952 0loss/FBt2/detection_loss‚0.36781243873408165 epoch‚45 2loss/FBt2/assignment_loss‚0.008121382130413171 1loss/SRqqt1/detection_loss‚0.8485626681457178 /loss/FBt1/detection_loss‚0.4857933771927492 0loss/FRt1/assignment_loss‚1.4325556070981975 0loss/FRt2/assignment_loss‚0.7281902821318618 3loss/SRqqt1/assignment_loss‚0.46596688869477587 /loss/SRqqt2/assignment_loss‚0.4172780202163 .loss/FRt2/detection_loss‚1.123711122084834 $loss/total_loss‚6.99161946358435 /loss/FRt1/detection_loss‚0.5470919953356665 trainer/global_step‚18349 % _timestamp‚1.7843560608349233e+09 _runtime‚15719.528231653 _step‚1512Xû¡Y§8Š /loss/FRt1/detection_loss‚0.5470919953356665 trainer/global_step‚18349 % _timestamp‚1.7843560608349233e+09 _runtime‚15719.528231653 _step‚1512 3loss/FBt1/assignment_loss‚3.228578565976174e-07 1loss/SRqqt2/detection_loss‚0.5665353589618952 0loss/FBt2/detection_loss‚0.36781243873408165 epoch‚45 2loss/FBt2/assignment_loss‚0.008121382130413171 1loss/SRqqt1/detection_loss‚0.8485626681457178 /loss/FBt1/detection_loss‚0.4857933771927492 0loss/FRt1/assignment_loss‚1.4325556070981975 0loss/FRt2/assignment_loss‚0.7281902821318618 3loss/SRqqt1/assignment_loss‚0.46596688869477587 /loss/SRqqt2/assignment_loss‚0.4172780202163 .loss/FRt2/detection_loss‚1.123711122084834 $loss/total_loss‚6.99161946358435è ±ªë“¨8‚(:ˆ ݹìÒ© œ“ gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09 )4©8‚(:© ݹìÒ­ÃÉ– network.sent‚ 2.9460089e+07 disk.drbd2165.out‚0 disk.dm-0.in‚ 26607.2890625 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 49.2236328125 disk.rbd0.out‚0 cpu‚26.171639838897004 disk.dm-1.in‚ 233.44140625* proc.memory.percent‚17.083096504211426$ disk./.usageGB‚585.6303672790527 disk.sda1.out‚0$ memory_percent‚97.69079685211182 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚26621.32421875& disk.nvme0c0n1.out‚35072.279296875 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1533.5859375 network.recv‚ 4.184305e+06$ disk.nvme0n1.out‚35072.279296875 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625( proc.memory.availableMB‚ 378.33984375# proc.memory.rssMB‚ 2798.89453125 disk.dm-1.out‚ 1533.5859375! disk.dm-0.out‚35072.279296875 proc.cpu.threads‚17) disk./.usagePercent‚39.30214532470439 disk.sda.out‚ 1704.12109375 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚26621.32421875 disk.rbd0.in‚ 0.16015625ePÊmª8‚(:’ ì¹ìҊݸ“ gpu.0.gpu‚52.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÃåá4«8‚(:© ì¹ìÒµœ‰–) disk./.usagePercent‚39.30216887739754 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.68922328948975 network.recv‚ 4.188934e+06$ disk./.usageGB‚585.6307182312012 cpu‚26.17120862463626 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1533.78125( disk.nvme0c0n1.out‚35075.32958984375" disk.nvme0n1.in‚26732.83203125 disk.rbd1.out‚ 49.2236328125 disk.drbd2165.in‚ 2.25390625 disk.rbd0.in‚ 0.16015625 disk.sda.in‚541.0478515625( proc.memory.availableMB‚ 378.59765625# proc.memory.rssMB‚ 2798.89453125 proc.cpu.threads‚17 network.sent‚ 2.9523831e+07 disk.rbd0.out‚0 disk.sda.out‚ 1704.31640625 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26732.83203125 disk.dm-0.in‚ 26718.796875 disk.sda3.out‚ 1533.78125 disk.sda1.out‚0# disk.dm-0.out‚35075.32958984375& disk.nvme0n1.out‚35075.32958984375* proc.memory.percent‚17.083096504211426óÊ⓬8‚(:ˆ û¹ìÒ ¸¹“ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/^»0­8‚(:¥ û¹ìÒ½’ç• disk.sda.in‚541.0478515625# proc.memory.rssMB‚ 2798.89453125! disk.dm-0.out‚35080.662109375 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚35080.662109375 disk.rbd1.in‚ 2.1796875$ disk./.usageGB‚585.6306381225586 disk.sda1.in‚ 8.759765625 disk.sda3.in‚ 417.107421875 network.sent‚ 2.9528456e+07 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26732.83203125" disk.nvme0n1.in‚26732.83203125 disk.dm-1.out‚ 1534.12890625 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1704.6640625 disk.sda3.out‚ 1534.12890625 disk.dm-0.in‚ 26718.796875# memory_percent‚97.6907730102539) disk./.usagePercent‚39.30216350123932 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625& disk.nvme0c0n1.out‚35080.662109375 disk.sda2.out‚ 170.53515625 proc.cpu.threads‚17 network.recv‚ 4.192257e+06 disk.drbd2165.out‚0 disk.rbd1.out‚ 49.2861328125% proc.memory.availableMB‚ 378.34375* proc.memory.percent‚17.083096504211426 cpu‚26.170793324326375 disk.rbd0.out‚0]ç[ 4®8/ _wandbruntime‚15758  _runtime‚15758W{0ʯ8Ä % _timestamp‚1.7843561000556526e+09 _runtime‚15758.749321347 _step‚1513 & lr-AdamW/pg1‚7.352693333371187e-06 & lr-AdamW/pg2‚7.352693333371187e-06 trainer/global_step‚18399ÜÞbbÏ°8É & lr-AdamW/pg1‚7.352693333371187e-06 & lr-AdamW/pg2‚7.352693333371187e-06 trainer/global_step‚18399 % _timestamp‚1.7843561000556526e+09 _runtime‚15758.749321347 _step‚1513é AµÆã4±8/ _wandbruntime‚15758  _runtime‚15758휟³ &²8P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145862632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615929180284928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19664955228672 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2781721490894848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39530599 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2781721473855488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145862630 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19425186008001 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2815556156749312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316272290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71757609 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635594116901376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615929161672704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2796101285478400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32227278 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2815555798370816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚15758.750527454 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2796101268439040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2796101285478400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316262311 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2796101268439040 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2815556156749312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71757880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19424845458111 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19454871270912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2815555798370816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚þ´ÿü2801146676902849 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2801146676902849 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19664955228672 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2781721490894848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39530599 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2781721473855488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145862630 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19425186008001 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2815556156749312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316272290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71757609 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635594116901376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615929161672704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2796101285478400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32227278 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2815555798370816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚15758.750527454 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2796101268439040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2796101285478400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316262311 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144ê 7<_4´8/ _wandbruntime‚15759  _runtime‚15759.ÀÞ'µ8‹N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19455542558720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2781868001566720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843561006722672e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚615962590431232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2801293841039193 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2815704052027392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71761664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170408733 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635628581114880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2815704410413056 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2796248526508032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170418724 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19426197061835 trainer/global_step‚18399 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32229023 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316288988 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145870262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316288988 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19665990683648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71761408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2796248509468672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316278995 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19666075059200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1515 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19455883905024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316278995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2815704410413056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19455883905024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170408733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635628684102656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚15759.365892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32229026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145870262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39532638 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19425856511833 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145870264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32229023 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316288988 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145870262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316288988 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19665990683648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71761408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2796248509468672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316278995 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19666075059200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1515 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19455883905024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316278995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2815704410413056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19455883905024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170408733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635628684102656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 _runtime‚15759.365892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32229026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145870262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39532638 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19425856511833 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145870264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2801293841039193 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2815704052027392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71761664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170408733 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635628581114880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438ë à%Ú4·8/ _wandbruntime‚15759  _runtime‚15759$"A¸8‹ 1loss/SRqqt1/detection_loss‚0.9050476807552297 3loss/SRqqt1/assignment_loss‚0.31337012327035874 /loss/FBt1/detection_loss‚0.3763503899130488 0loss/FBt2/detection_loss‚0.36850031427730895 %loss/total_loss‚6.770335312971204 epoch‚45 trainer/global_step‚18399 _step‚1516 3loss/SRqqt2/assignment_loss‚0.40091953644151773 2loss/FBt1/assignment_loss‚0.000906345296799671 3loss/FBt2/assignment_loss‚8.422229411489358e-05 /loss/FRt2/detection_loss‚1.1266550390797063 % _timestamp‚1.7843561006744988e+09 0loss/FRt1/assignment_loss‚1.4490235870967276 0loss/FRt2/assignment_loss‚0.7139295098746394 /loss/FRt1/detection_loss‚0.5600513390440187 1loss/SRqqt2/detection_loss‚0.5554972256277336 _runtime‚15759.367774377bVe¹8 trainer/global_step‚18399 _step‚1516 3loss/SRqqt2/assignment_loss‚0.40091953644151773 2loss/FBt1/assignment_loss‚0.000906345296799671 3loss/FBt2/assignment_loss‚8.422229411489358e-05 /loss/FRt2/detection_loss‚1.1266550390797063 % _timestamp‚1.7843561006744988e+09 0loss/FRt1/assignment_loss‚1.4490235870967276 0loss/FRt2/assignment_loss‚0.7139295098746394 /loss/FRt1/detection_loss‚0.5600513390440187 1loss/SRqqt2/detection_loss‚0.5554972256277336 _runtime‚15759.367774377 1loss/SRqqt1/detection_loss‚0.9050476807552297 3loss/SRqqt1/assignment_loss‚0.31337012327035874 /loss/FBt1/detection_loss‚0.3763503899130488 0loss/FBt2/detection_loss‚0.36850031427730895 %loss/total_loss‚6.770335312971204 epoch‚45ì Ü?§›º8‚(:ˆ ŠºìҀǕ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚169.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.po)‰¶çwerPercent‚33.8662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#7½2»8‚(:§ ŠºìÒ´›˜ disk.dm-0.in‚ 26718.796875# proc.memory.rssMB‚ 2800.20703125) proc.memory.percent‚17.09110736846924 network.recv‚ 4.195346e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 proc.cpu.threads‚17 disk.dm-1.in‚ 233.44140625) disk./.usagePercent‚39.30217399754823 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625! disk.dm-0.out‚35084.376953125$ disk.nvme0n1.out‚35084.376953125 disk.rbd1.out‚ 49.2861328125( proc.memory.availableMB‚ 377.12109375 network.sent‚ 2.9533563e+07 disk.sda.out‚ 1705.94921875 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1535.4140625 disk.rbd0.out‚0 disk.dm-1.out‚ 1535.4140625 disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚26732.83203125$ memory_percent‚97.69823551177979 cpu‚26.171746711237017 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚26732.83203125$ disk./.usageGB‚585.6307945251465& disk.nvme0c0n1.out‚35084.376953125t²x4¼8/ _wandbruntime‚15776  _runtime‚15776êj¶Ð{=½8õz 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2780464291572571 jet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.04078979045152664 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23851776123046875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33766600489616394 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6564521193504333 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 )jet/accuracy_1_of_1‚0.345990389585495 jet/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2742655873298645 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12325631082057953 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803701996803284 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6259750127792358 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2825920283794403 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6333988904953003 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2338828295469284 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7233903408050537 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7243085503578186 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6374274492263794 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8306500315666199 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_5‚1 +jet/accuracy_2_of_2‚0.12623463571071625 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2400781363248825 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5333369374275208 $ _timestamp‚1.784356117804502e+09 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09529467672109604 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7275471091270447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15322323143482208 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6912482380867004 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24463921785354614 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5491742491722107 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_3‚0.9277561902999878 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6867591738700867 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23618987202644348 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_3_of_4‚0.9968254566192627 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03613817319273949 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6557325720787048 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7936913371086121 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2378978729248047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7794028520584106 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8984000086784363 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7328080534934998 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31466934084892273 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3627742826938629 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.774394690990448 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10066645592451096 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5678157210350037 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14098966121673584 *jet/accuracy_2_of_3‚0.5117545127868652 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚7.711531536127963 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9806751012802124 (particle/accuracy‚0.8331550359725952 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.786685585975647 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.626825749874115 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6110005378723145 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _step‚1517 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24790994822978973 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6673785448074341 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943199872970581 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.767423152923584 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7234129905700684 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.089603491127491 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 'particle/f_score‚0.6048258543014526 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.5945839881896973 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13665221631526947 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7080016136169434 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.792143702507019 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7951432466506958 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32921603322029114 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739060640335083 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.653666615486145 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 epoch‚45 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 +jet/accuracy_3_of_3‚0.03792091831564903 6validation_average_jet_accuracy‚0.3791799545288086 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6901756525039673 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6525381803512573 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.56636643409729 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37586483359336853 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7160643935203552 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 trainer/global_step‚18399 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24913987517356873 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03636068105697632 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04629756137728691 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18415594100952148 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 *jet/accuracy_1_of_2‚0.7863299250602722 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2225669026374817 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7838338017463684 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15069782733917236 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6362293362617493 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7278417944908142 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05191571265459061 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29577258229255676 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28224751353263855 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02569410391151905 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2825920283794403 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025763902813196182 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6687365174293518 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6845487356185913 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚15776.498758368 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7097358107566833 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7638329267501831 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510194480419159 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6362293362617493 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21211281418800354 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21279487013816833 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2825976610183716 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9452828764915466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8355631232261658 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02553623542189598 *jet/accuracy_2_of_4‚0.7723941802978516 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29328522086143494 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025511378422379494 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29306560754776 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16748867928981781 W:l=¾8úz 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7160643935203552 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 trainer/global_step‚18399 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24913987517356873 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03636068105697632 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04629756137728691 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18415594100952148 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 *jet/accuracy_1_of_2‚0.7863299250602722 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2225669026374817 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7838338017463684 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15069782733917236 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6362293362617493 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7278417944908142 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05191571265459061 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29577258229255676 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28224751353263855 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02569410391151905 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2825920283794403 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025763902813196182 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6687365174293518 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6845487356185913 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚15776.498758368 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7097358107566833 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7638329267501831 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2510194480419159 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6362293362617493 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21211281418800354 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21279487013816833 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2825976610183716 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9452828764915466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8355631232261658 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02553623542189598 *jet/accuracy_2_of_4‚0.7723941802978516 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29328522086143494 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025511378422379494 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29306560754776 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1674886792898178 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2780464291572571 jet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.04078979045152664 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23851776123046875 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33766600489616394 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6564521193504333 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 )jet/accuracy_1_of_1‚0.345990389585495 jet/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2742655873298645 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12325631082057953 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8803701996803284 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6259750127792358 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2825920283794403 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6333988904953003 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2338828295469284 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7233903408050537 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7243085503578186 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6374274492263794 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8306500315666199 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_5‚1 +jet/accuracy_2_of_2‚0.12623463571071625 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2400781363248825 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5333369374275208 $ _timestamp‚1.784356117804502e+09 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09529467672109604 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7275471091270447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15322323143482208 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6912482380867004 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24463921785354614 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5491742491722107 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_3‚0.9277561902999878 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6867591738700867 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23618987202644348 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_3_of_4‚0.9968254566192627 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03613817319273949 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6557325720787048 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7936913371086121 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2378978729248047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7794028520584106 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8984000086784363 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7328080534934998 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31466934084892273 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3627742826938629 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.774394690990448 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10066645592451096 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5678157210350037 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14098966121673584 *jet/accuracy_2_of_3‚0.5117545127868652 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚7.711531536127963 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9806751012802124 (particle/accuracy‚0.8331550359725952 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.786685585975647 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8696901202201843 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.626825749874115 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6110005378723145 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _step‚1517 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24790994822978973 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6673785448074341 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.943199872970581 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.767423152923584 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7234129905700684 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.089603491127491 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 'particle/f_score‚0.6048258543014526 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.5945839881896973 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13665221631526947 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7080016136169434 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.792143702507019 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7951432466506958 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32921603322029114 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739060640335083 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.653666615486145 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 epoch‚45 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 +jet/accuracy_3_of_3‚0.03792091831564903 6validation_average_jet_accuracy‚0.3791799545288086 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6901756525039673 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6525381803512573 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.56636643409729 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37586483359336853 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9ú`˜×970238208770752í Ädpzž¿8‚(:“ ™ºìÒ˜¹µ“ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í#Ë2À8‚(:§ ™ºìÒÅàî• disk.sda1.in‚ 8.759765625$ disk.nvme0c0n1.in‚26732.83203125 disk.dm-0.in‚ 26718.796875 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 49.3486328125 disk.dm-1.in‚ 233.44140625" disk.nvme0n1.in‚26732.83203125 proc.cpu.threads‚17 network.sent‚ 2.9596823e+07 network.recv‚ 4.199858e+06$ disk./.usageGB‚585.6302146911621 disk.rbd0.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2803.171875 cpu‚26.174185263783187) disk./.usagePercent‚39.30213508440302 disk.sda2.out‚ 170.53515625 disk.drbd2165.out‚0$ memory_percent‚97.71528244018555 disk.sda.out‚ 1706.60546875 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875* proc.memory.percent‚17.109203338623047" disk.dm-0.out‚35089.7099609375 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1536.0703125& proc.memory.availableMB‚ 374.328125 disk.sda.in‚541.0478515625% disk.nvme0n1.out‚35089.7099609375 disk.drbd2165.in‚ 2.25390625' disk.nvme0c0n1.out‚35089.7099609375 disk.sda3.out‚ 1536.0703125µ½g‡“Á8‚(:ˆ ¨ºìÒ̪͓ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ö&»1Â8‚(:¦ ¨ºìÒŠœ–$ disk.nvme0c0n1.in‚26869.51171875" disk.nvme0n1.in‚26869.51171875 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.dm-0.in‚ 26855.4765625 disk.sda.out‚ 1706.96484375 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1536.4296875 network.sent‚ 2.9654052e+07 disk.rbd1.out‚ 49.3486328125 disk.sda1.in‚ 8.759765625 disk.rbd0.in‚ 0.16015625* proc.memory.percent‚17.109203338623047 cpu‚26.173750346021308# disk./.usageGB‚585.622615814209 disk.sda.in‚541.0478515625 disk.sda1.out‚0( proc.memory.availableMB‚ 373.75390625) disk./.usagePercent‚39.30162511739482& disk.nvme0c0n1.out‚35093.189453125! disk.dm-0.out‚35093.189453125 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚35093.189453125$ memory_percent‚97.71878719329834! proc.memory.rssMB‚ 2803.171875 disk.drbd2165.in‚ 2.25390625 proc.cpu.threads‚17 network.recv‚ 4.203659e+06 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.dm-1.out‚ 1536.4296875 disk.drbd2165.out‚0T«R+žÃ8‚(:“ ·ºìÒ”Œª“ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.307' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.861399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ÀöK6Ä8‚(:« ·ºìÒÙÊÓ• disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.nvme0n1.in‚26869.51171875! proc.memory.rssMB‚ 2803.171875* proc.memory.percent‚17.109203338623047( disk.nvme0c0n1.out‚35100.47509765625& disk.nvme0n1.out‚35100.47509765625 disk.sda3.out‚ 1536.7265625 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.dm-0.in‚ 26855.4765625# disk.dm-0.out‚35100.47509765625 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875' proc.memory.availableMB‚ 373.8046875 cpu‚26.17333519908229 network.sent‚ 2.9659155e+07$ disk./.usageGB‚585.6226005554199 disk.dm-1.in‚ 233.44140625 disk.sda.out‚ 1707.26171875 disk.rbd1.out‚ 49.4111328125$ memory_percent‚97.71847724914551 disk.sda1.in‚ 8.759765625$ disk.nvme0c0n1.in‚26869.51171875 disk.rbd1.in‚ 2.1796875 disk.drbd2165.in‚ 2.25390625 network.recv‚ 4.207106e+06) disk./.usagePercent‚39.30162409336468 disk.sda.in‚541.0478515625 disk.dm-1.out‚ 1536.72656251Jâ‡4Å8/ _wandbruntime‚15815  _runtime‚15815,t ÊÆ8Ä _runtime‚15815.982722462 _step‚1518 & lr-AdamW/pg1‚6.902939666177184e-06 & lr-AdamW/pg2‚6.902939666177184e-06 trainer/global_step‚18449 % _timestamp‚1.7843561572890902e+090cdhÏÇ8É _runtime‚15815.982722462 _step‚1518 & lr-AdamW/pg1‚6.902939666177184e-06 & lr-AdamW/pg2‚6.902939666177184e-06 trainer/global_step‚18449 % _timestamp‚1.7843561572890902e+09î ±ø‡44È8/ _wandbruntime‚15815  _runtime‚15815{Þ4(É8ÿO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2790955670663168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2805392886571008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32317048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617616511698944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170934432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170924455 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2824916203588608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19734479729152 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2790955687702528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170934432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19734399540736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71983221 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2824916203588608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71982945 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19523299978240 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784356157290349e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32317051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19493181670248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2824915845210112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1519 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19523299978240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317299330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19493522220138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146364898 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146364898 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚637351010040320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2810448852333416 trainer/global_step‚18449 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170924455 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19522958639104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19522958639104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚637350911239680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617616530311168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2824915845210112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2824915845210112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71983221 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2824916203588608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71982945 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19523299978240 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784356157290349e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32317051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19493181670248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2824915845210112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1519 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19523299978240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317299330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19493522220138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146364898 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146364898 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚637351010040320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2810448852333416 trainer/global_step‚18449 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170924455 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19522958639104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19522958639104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚637350911239680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617616530311168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317316028 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19494533273972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170943498 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825064457252352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71987061 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2825064457252352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2805540144640000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170933507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32318796 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71986801 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317306035 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚637385466433536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617649959069696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39668008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2805540127600640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2825064098866688 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2805540144640000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146372528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19523971266048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146372528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19523971266048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317316028 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19494533273972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170943498 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825064457252352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71987061 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2825064457252352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2805540144640000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170933507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32318796 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71986801 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317306035 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚637385466433536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617649959069696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39668008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2805540127600640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2825064098866688 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2805540144640000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32404301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171378907 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171388884 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚18499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39770866 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639073854247424 $ _timestamp‚1.784356197434343e+09 _runtime‚15856.128057475 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72175429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2817825360003174 _step‚1523 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2812754955091968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19573654140928 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2832328950572032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171378907 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2832328592193536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2798281204256768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619287949625344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19573995480064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146746496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19573654140928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2798281221296128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2812754938052608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2812754955091968ìKCé‚ Ü8„P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2812754955091968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚619287968237568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19785886009856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19573995480064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32404301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171378907 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171388884 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚18499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39770866 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639073854247424 $ _timestamp‚OžJ:ˆ1.784356197434343e+09 _runtime‚15856.128057475 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72175429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2817825360003174 _step‚1523 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2812754955091968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19573654140928 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2832328950572032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171378907 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2832328592193536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2798281204256768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619287949625344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19573995480064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146746496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19573654140928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2798281221296128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2812754938052608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ó JMŽ¯4Ý8/ _wandbruntime‚15856  _runtime‚15856„·o''Þ8ŠN WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2832477204235776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19786914521088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72178975 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19544809210878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318152080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19574666767872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146754128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2817972524139518 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2812902179082240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2812902196121600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39772932 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚639108212723712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1524 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318142087 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171387959 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2832476845850112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚619321396996096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19786834339840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19575008114176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146754128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2817972881728880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171387959 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72179244 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2832477204235776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2812902179082240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32406043 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146754130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171397950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32406046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39773198 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619321378383872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2798427714928640 % _timestamp‚1.7843561980495408e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19545149760880 trainer/global_step‚18499 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146754130 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639108311517184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171397950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2]ä“'ß8N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32406046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39773198 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619321378383872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2798427714928640 % _timestamp‚1.7843561980495408e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19545149760880 trainer/global_step‚18499 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146754130 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639108311517184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171397950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2832477204235776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19786914521088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72178975 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19544809210878 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318152080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19574666767872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146754128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2817972524139518 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2812902179082240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2812902196121600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39772932 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚639108212723712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1524 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318142087 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171387959 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2832476845850112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚619321396996096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19786834339840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19575008114176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146754128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2817972881728880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171387959 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72179244 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2832477204235776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2812902179082240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32406043 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146754130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171397950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976ô Çý÷"4à8/ _wandbruntime‚15856  _runtime‚15856=b²ä á8‡ epoch‚46 2loss/SRqqt1/assignment_loss‚0.3922057278248425 4loss/FBt1/assignment_loss‚1.8294972805440086e-05 0loss/SRqqt1/detection_loss‚0.808062099200026 1loss/SRqqt2/detection_loss‚0.6057856901491694 _step‚1525 0loss/FRt2/assignment_loss‚0.6606052620924981 trainer/global_step‚18499 /loss/FBt2/detection_loss‚0.4467033254444659 &loss/total_loss‚6.8511434722603255 /loss/FRt1/assignment_loss‚1.471524168227426 2loss/SRqqt2/assignment_loss‚0.4224614135800212 1loss/FBt2/assignment_loss‚0.02455905423476163 /loss/FRt2/detection_loss‚1.1272268376725796 _runtime‚15856.74521301 /loss/FRt1/detection_loss‚0.5149277322302109 0loss/FBt1/detection_loss‚0.37706386663151836 % _timestamp‚1.7843561980518942e+09s6»€â8Œ 2loss/SRqqt2/assignment_loss‚0.4224614135800212 1loss/FBt2/assignment_loss‚0.02455905423476163 /loss/FRt2/detection_loss‚1.1272268376725796 _runtime‚15856.74521301 /loss/FRt1/detection_loss‚0.5149277322302109 0loss/FBt1/detection_loss‚0.37706386663151836 % _timestamp‚1.7843561980518942e+09 epoch‚46 2loss/SRqqt1/assignment_loss‚0.3922057278248425 4loss/FBt1/assignment_loss‚1.8294972805440086e-05 0loss/SRqqt1/detection_loss‚0.808062099200026 1loss/SRqqt2/detection_loss‚0.6057856901491694 _step‚1525 0loss/FRt2/assignment_loss‚0.6606052620924981 trainer/global_step‚18499 /loss/FBt2/detection_loss‚0.4467033254444659 &loss/total_loss‚6.8511434722603255 /loss/FRt1/assignment_loss‚1.471524168227426õ 3.Ú^“ã8‚(:ˆ óºìÒ¾­“ gpu.0.gpu‚87.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Cá3ä8‚(:¨ óºìÒ۾ߕ disk.dm-0.in‚26990.48828125 disk.sda1.out‚0 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1537.86328125# memory_percent‚97.7177619934082# disk.nvme0c0n1.in‚ 27004.5234375 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 27004.5234375* proc.memory.percent‚17.109203338623047) disk./.usagePercent‚39.30146767276126 disk.rbd1.out‚ 49.5986328125 disk.rbd0.out‚0 disk.dm-1.out‚ 1537.86328125# disk.dm-0.out‚35117.69091796875 disk.sda.in‚541.0478515625& disk.nvme0n1.out‚35117.69091796875! proc.memory.rssMB‚ 2803.171875$ disk./.usageGB‚585.6202697753906 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625( disk.nvme0c0n1.out‚35117.69091796875 disk.sda2.in‚ 82.544921875& proc.memory.availableMB‚ 373.921875 cpu‚26.17159575463286 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625 network.sent‚ 2.973665e+07 disk.drbd2165.out‚0 disk.sda.out‚ 1708.3984375 proc.cpu.threads‚17 network.recv‚ 4.221099e+06 disk.dm-1.in‚ 233.44140625&ls,å8‚(:’ ‚»ìÒÙÁÉ“ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.811' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.76219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÚ-^1æ8‚(:¦ ‚»ìÒ¤Þ——! disk.nvme0n1.in‚ 27004.5234375 disk.rbd1.out‚ 49.5986328125 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 27004.5234375 network.sent‚ 2.9800239e+07 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625' disk.nvme0c0n1.out‚35120.9013671875 cpu‚26.171177682108713 disk.sda3.in‚ 417.107421875 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625% disk.nvme0n1.out‚35120.9013671875( proc.memory.availableMB‚ 373.87890625 proc.cpu.threads‚17 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0) disk./.usagePercent‚39.30149890568043# disk./.usageGB‚585.620735168457 disk.sda3.out‚ 1538.328125 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1538.328125! proc.memory.rssMB‚ 2803.171875* proc.memory.percent‚17.109203338623047 disk.sda.out‚ 1708.86328125 disk.sda2.in‚ 82.544921875 disk.dm-0.in‚26990.48828125" disk.dm-0.out‚35120.9013671875$ memory_percent‚97.71802425384521 network.recv‚ 4.224947e+06 disk.sda.in‚541.0478515625¨œ’4ç8/ _wandbruntime‚15895  _runtime‚15895$ ÏëÅè8¿ $ lr-AdamW/pg1‚6.0453952913658e-06 $ lr-AdamW/pg2‚6.0453952913658e-06 trainer/global_step‚18549 $ _timestamp‚1.784356237285826e+09 _runtime‚15895.979408948 _step‚1526É÷ Êé8Ä $ lr-AdamW/pg1‚6.0453952913658e-06 $ lr-AdamW/pg2‚6.0453952913658e-06 trainer/global_step‚18549 $ _timestamp‚1.784356237285826e+09 _runtime‚15895.979408948 _step‚1526ö (¶ù!4ê8/ _wandbruntime‚15895  _runtime‚15895¦„GrÑë8P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147128096 JDDeviceStatsMonitor.on_train_batchZÎœr6'_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820117006573568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2839741697555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39875655 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839741697555456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318961455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171833359 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820116989534208 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839741339176960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318971434 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32491548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19624690981888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171833359 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839741339176960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820117006573568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171843336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19594414644064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101ëÅGÓ (ì8†P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147128098 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620959387551744 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19837076076032 _runtime‚15895.980653107 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19837170944512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839741697555456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318961455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171833359 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820116989534208 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839741339176960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318971434 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32491548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19624690981888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171833359 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839741339176960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820117006573568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171843336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19594414644064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147128096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820117006573568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2839741697555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39875655 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318988132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318988132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39878003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620992834922496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19595425697786 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2839889951219200 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2820264230563840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171852402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19595766247788 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171842411 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839889951219200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19625703616000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19625362269696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2805753248522240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640830925213696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171842411 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19625703616000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640831026104320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19838191181824 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2825348674220026 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318978139 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7843562378992133e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820264247603200 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2839889592833536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32493296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72371055 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19838108903424 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2825349031809388 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171852402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39877762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1528S1*V ï8N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620992816310272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72371055 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19838108903424 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2825349031809388 OADeviceStatsMonitor.on_train)Eë‘À_batch_end/active.small_pool.allocated‚ 171852402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39877762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147135728 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2820264230563840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318988132 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318988132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39878003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620992834922496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19595425697786 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2839889951219200 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2820264230563840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171852402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19595766247788 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171842411 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839889951219200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19625703616000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19625362269696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2805753248522240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640830925213696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171842411 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19625703616000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640831026104320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19838191181824 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2825348674220026 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318978139 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7843562378992133e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820264247603200 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2839889592833536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32493296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2847154086160384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847154086160384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630825478144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚1531 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19675045144576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2832577660164190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19645371680862 $ _timestamp‚1.784356277318446e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚642519303690240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172287811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚15936.012139299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2847154444538880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147509696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19888560497664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚18599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630844090368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2832577302574940 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319807486 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2812932288483328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147509698 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19888478212096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39980712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642519404588032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72559513 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39980712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642519404588032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72559513 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2847154086160384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847154086160384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630825478144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚1531 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19675045144576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2832577660164190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19645371680862 $ _timestamp‚1.784356277318446e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_%/fòstart/inactive_split_bytes.all.freed‚642519303690240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172287811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚15936.012139299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2847154444538880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147509696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19888560497664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚18599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630844090368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2832577302574940 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976û >Òpu4ÿ8/ _wandbruntime‚15936  _runtime‚15936 8Ùã'€9‹N XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2832725181889896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172306854 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72563072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32580543 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19646042184694 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19676399117824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19889585573888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2847302698202624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72563333 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19676399117824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2827626282045440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172296863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2832724824300534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172306854 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847302698202624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319824184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319814191 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39982787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622664272848896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147517330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19889499101184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642553858422784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2813078799155200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39982529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642553753337856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19646382734696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2827626299084800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147517328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0fn§ú'9N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147517328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2832725181889896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172306854 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72563072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32580543 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19646042184694 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19676399117824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19889585573888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2847302698202624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72563333 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19676399117824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2827626282045440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172296863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2832724824300534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172306854 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847302698202624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319824184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319814191 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39982787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622664272848896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147517330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19889499101184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642553858422784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2813078799155200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39982529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642553753337856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19646382734696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2827626299084800ü ®|è+4‚9/ _wandbruntime‚15936  _runtime‚15936¤¸Iqƒ9‹ .loss/FRt2/detection_loss‚1.101277078274503 % _timestamp‚1.7843562779361396e+09 _step‚1533 0loss/FRt2/assignment_loss‚0.6739580160573153 /loss/FRt1/detection_loss‚0.5034028736122285 1loss/SRqqt1/detection_loss‚0.8024001664621613 /loss/FBt1/detection_loss‚0.6235625493118278 trainer/global_step‚18599 _runtime‚15936.629451081 0loss/FRt1/assignment_loss‚1.5000368722186965 3loss/SRqqt1/assignment_loss‚0.44654178233593717 3loss/FBt1/assignment_loss‚3.350539576874034e-05 0loss/FBt2/detection_loss‚0.40284846276966196 epoch‚46 3loss/SRqqt2/assignment_loss‚0.37283092996451034 3loss/FBt2/assignment_loss‚5.601784702312175e-05 1loss/SRqqt2/detection_loss‚0.5238481594628678 %loss/total_loss‚6.950796413712501ûä3A„9 epoch‚46 3loss/SRqqt2/assignment_loss‚0.37283092996451034 3loss/FBt2/assignment_loss‚5.601784702312175e-05 1loss/SRqqt2/detection_loss‚0.5238481594628678 %loss/total_loss‚6.950796413712501 .loss/FRt2/detection_loss‚1.101277078274503 % _timestamp‚1.7843562779361396e+09 _step‚1533 0loss/FRt2/assignment_loss‚0.6739580160573153 /loss/FRt1/detection_loss‚0.5034028736122285 1loss/SRqqt1/detection_loss‚0.8024001664621613 /loss/FBt1/detection_loss‚0.6235625493118278 trainer/global_step‚18599 _runtime‚15936.629451081 0loss/FRt1/assignment_loss‚1.5000368722186965 3loss/SRqqt1/assignment_loss‚0.44654178233593717 3loss/FBt1/assignment_loss‚3.350539576874034e-05 0loss/FBt2/detection_loss‚0.40284846276966196ý þ/vž…9‚(:“ ¾»ìÒðýé“ gpu.0.gpu‚38.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.900400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓOÎc0†9‚(:¥ ¾»ìÒ¶…ø• disk.dm-0.in‚ 27126.8359375 disk.dm-1.out‚ 1539.74609375$ disk.nvme0c0n1.in‚27140.87109375( proc.memory.availableMB‚ 373.51953125! proc.memory.rssMB‚ 2803.171875 disk.sda.out‚ 1710.28125 disk.rbd1.in‚ 2.1796875$ memory_percent‚97.72021770477295 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625& disk.nvme0c0n1.out‚35141.435546875 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚35141.435546875 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚35141.435546875 network.sent‚ 2.9878832e+07$ disk./.usageGB‚585.6295890808105 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625* proc.memory.percent‚17.109203338623047 cpu‚26.16944344895039 network.recv‚ 4.239235e+06) disk./.usagePercent‚39.30209309916741 disk.rbd0.in‚ 0.16015625 disk.sda3.out‚ 1539.74609375 disk.rbd1.out‚ 49.7236328125 disk.sda.in‚541.0478515625" disk.nvme0n1.in‚27140.87109375 disk.sda3.in‚ 417.107421875wW¾Z“‡9‚(:ˆ Í»ìÒ°Ë“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø'2ˆ9‚(:§ Í»ìÒѲ˜— network.sent‚ 2.9941902e+07 disk.dm-1.out‚ 1540.08984375 disk.drbd2165.in‚ 2.25390625# disk.dm-0.out‚35147.33544921875& disk.nvme0n1.out‚35147.33544921875 disk.rbd1.out‚ 49.7861328125! proc.memory.rssMB‚ 2803.171875 disk.dm-1.in‚ 233.44140625$ memory_percent‚97.72062301635742* proc.memory.percent‚17.109203338623047 cpu‚26.16901438458237 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.dm-0.in‚ 27126.8359375 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1540.08984375 disk.rbd0.in‚ 0.16015625 disk.sda.out‚1710.625 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27140.87109375 disk.sda3.in‚ 417.107421875 proc.cpu.threads‚17) disk./.usagePercent‚39.30209770730303$ disk.nvme0c0n1.in‚27140.87109375 disk.sda2.in‚ 82.544921875& proc.memory.availableMB‚ 373.453125$ disk./.usageGB‚585.6296577453613 network.recv‚ 4.243946e+06 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚35147.33544921875 disk.rbd1.in‚ 2.17968752~y‰9‚(:’ Ü»ìÒÝ’Ê“ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04þS4Š9‚(:© Ü»ìÒ¹–ו* proc.memory.percent‚17.109203338623047 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 27293.2734375' disk.nvme0c0n1.out‚35150.0517578125$ memory_percent‚97.71926403045654 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1540.42578125! proc.memory.rssMB‚ 2803.171875 network.sent‚ 2.9946459e+07 disk.drbd2165.in‚ 2.25390625! disk.nvme0n1.in‚ 27293.2734375 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 49.7861328125 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1540.42578125" disk.dm-0.out‚35150.0517578125( proc.memory.availableMB‚ 373.67578125 cpu‚26.16857995302278 disk.sda1.out‚0 disk.sda2.out‚ 170.53515625% disk.nvme0n1.out‚35150.0517578125 disk.rbd1.in‚ 2.1796875 network.recv‚ 4.247115e+06) disk./.usagePercent‚39.30209745129549$ disk./.usageGB‚585.6296539306641 disk.dm-1.in‚ 233.44140625 disk.dm-0.in‚27279.23828125 disk.sda.out‚ 1710.9609375äZY>4‹9/ _wandbruntime‚15976  _runtime‚15976FðÊŒ9Ä & lr-AdamW/pg1‚5.243983623160334e-06 & lr-AdamW/pg2‚5.243983623160334e-06 trainer/global_step‚18649 % _timestamp‚1.7843563173711324e+09 _runtime‚15976.064689609 _step‚1534û©ÑÏ9É _runtime‚15976.064689609 _step‚1534 & lr-AdamW/pg1‚5.243983623160334e-06 & lr-AdamW/pg2‚5.243983623160334e-06 trainer/global_step‚18649 % _timestamp‚1.7843563173711324e+09þ ¹k~j4Ž9/ _wandbruntime‚15976  _runtime‚15976œÞm–u9ÿO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2834841109536768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2839953810244698 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320633559 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172742263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2839953452655448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172752240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320643538 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19726081985536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚18649 _step‚1535 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72750706 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2834841092497408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.smaM Dll_pool.allocated‚40084655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2854567191522304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19695647617880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147891298 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147891296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2854566833143808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72750446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624302282016768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19939889798656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2854566833143808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644242171815424 _runtime‚15976.065880367 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2834841092497408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19725740646400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2834841092497408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19725740646400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19726081985536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚18649 _step‚1535 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72750706 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2834841092497408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40084655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2854567191522304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19695647617880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147891298 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147891296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2854566833143808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72750446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624302282016768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19939889798656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2854566833143808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269ÿ ¥9ãƒ4‘9/ _wandbruntime‚15976  _runtime‚15976–|_ž'’9ŠN H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19940922330624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19696658671602 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644276633105920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644276536409600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320650243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19696999221604 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32667793 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147898928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19726753273344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19726753273344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624335710775296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2840101331970404 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172761306 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2854715445186048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320650243 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147898930 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2840100974381042 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843563179866128e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19727094619648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2834988350566400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854715445186048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624335692163072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2820404332748800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚18649 _step‚1536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚15976.680377889 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2854715086800384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19727094619648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72754279 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2834988333527040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172751315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2834988350566400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32667796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320660236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2834988333527040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360üs¾(ÿ“9N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19727094619648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72754279 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2834988333527040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172751315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2834988350566400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32667796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320660236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2834988333527040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19940922330624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19696658671602 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644276633105920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644276536409600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320650243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19696999221604 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32667793 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147898928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19726753273344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19726753273344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624335710775296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2840101331970404 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172761306 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2854715445186048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320650243 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147898930 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2840100974381042 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843563179866128e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19727094619648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2834988350566400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854715445186048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624335692163072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2820404332748800 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚18649 _step‚1536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚15976.680377889 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2854715086800384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154€ øÄ×4”9/ _wandbruntime‚15976  _runtime‚15976&.7•9ˆ epoch‚46 2loss/SRqqt2/assignment_loss‚0.4384610377332385 2loss/FBt2/assignment_loss‚0.010690434018760646 trainer/global_step‚18649 % _timestamp‚1.7843563179891016e+09 _runtime‚15976.682418822 1loss/SRqqt1/detection_loss‚0.8454090752259673 0loss/FBt2/detection_loss‚0.38717629874407755 %loss/total_loss‚7.216465160950213 0loss/FRt1/assignment_loss‚1.4059246496116045 0loss/FRt2/assignment_loss‚0.7508706156920669 3loss/FBt1/assignment_loss‚0.0010174586426014815 /loss/FRt1/detection_loss‚0.5406834227806114 0loss/SRqqt2/detection_loss‚0.559272919773745 /loss/FBt1/detection_loss‚0.6607978674672688 3loss/SRqqt1/assignment_loss‚0.44319252745625504 .loss/FRt2/detection_loss‚1.172968853804015 _step‚1537g kA–9 epoch‚46 2loss/SRqqt2/assignment_loss‚0.4384610377332385 2loss/FBt2/assignment_loss‚0.010690434018760646 trainer/global_step‚18649 % _timestamp‚1.7843563179891016e+09 _runtime‚15976.682418822 1loss/SRqqt1/detection_loss‚0.8454090752259673 0loss/FBt2/detection_loss‚0.38717629874407755 %loss/total_loss‚7.216465160950213 0loss/FRt1/assignment_loss‚1.4059246496116045 0loss/FRt2/assignment_loss‚0.7508706156920669 3loss/FBt1/assignment_loss‚0.0010174586426014815 /loss/FRt1/detection_loss‚0.5406834227806114 0loss/SRqqt2/detection_loss‚0.559272919773745 /loss/FBt1/detection_loss‚0.6607978674672688 3loss/SRqqt1/assignment_loss‚0.44319252745625504 .loss/FRt2/detection_loss‚1.172968853804015 _step‚1537 ˜¿}ê“—9‚(:ˆ ë»ìÒ¼³‚– gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž;¾!4˜9‚(:© ë»ìÒê „˜% disk.nvme0n1.out‚35154.6396484375 disk.sda3.in‚ 417.107421875 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0$ disk./.usageGB‚585.6298179626465 disk.dm-0.in‚27279.23828125 network.sent‚ 2.9951607e+07 network.recv‚ 4.250998e+06 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 27293.2734375 disk.sda3.out‚ 1540.98828125 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625( proc.memory.availableMB‚ 373.62109375) disk./.usagePercent‚39.30210845961946# disk.nvme0c0n1.in‚ 27293.2734375' disk.nvme0c0n1.out‚35154.6396484375 cpu‚26.16814875064328$ memory_percent‚97.71959781646729* proc.memory.percent‚17.109203338623047 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1540.98828125 disk.sda.in‚541.0478515625" disk.dm-0.out‚35154.6396484375 disk.sda.out‚ 1711.5234375 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 49.8486328125 disk.sda1.in‚ 8.759765625! proc.memory.rssMB‚ 2803.171875 proc.cpu.threads‚17¯™9‚(:… ú»ìÒˆ¡¾“ gpu.0.gpu‚68.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚195.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.11 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dæ·Ô4š9‚(:© ú»ìÒ“„—# disk.nvme0c0n1.in‚ 27293.2734375 network.sent‚ 3.0014906e+07 network.recv‚ 4.255113e+06 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1541.34765625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1541.34765625 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 49.8486328125' disk.nvme0c0n1.out‚35157.6220703125 disk.sda.out‚ 1711.8828125$ disk./.usageGB‚585.6298446655273 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875 disk.rbd0.out‚0! proc.memory.rssMB‚ 2803.171875 disk.dm-0.in‚27279.23828125" disk.dm-0.out‚35157.6220703125 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0* proc.memory.percent‚17.109203338623047 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625% disk.nvme0n1.out‚35157.6220703125 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0$ memory_percent‚97.72119522094727* disk./.usagePercent‚39.302110251672204! disk.nvme0n1.in‚ 27293.2734375& proc.memory.availableMB‚ 373.359375 cpu‚26.167729840901313 proc.cpu.threads‚17ûá¡34›9/ _wandbruntime‚16015  _runtime‚16015šŸ"HÊœ9Ä trainer/global_step‚18699 % _timestamp‚1.7843563571427972e+09 _runtime‚16015.836348742 _step‚1538 & lr-AdamW/pg1‚4.864392459259709e-06 & lr-AdamW/pg2‚4.864392459259709e-06H%hŸÏ9É & lr-AdamW/pg2‚4.864392459259709e-06 trainer/global_step‚18699 % _timestamp‚1.7843563571427972e+09 _runtime‚16015.836348742 _step‚1538 & lr-AdamW/pg1‚4.864392459259709e-06‚ ¦Ð†g4ž9/ _wandbruntime‚16015  _runtime‚16015Tœ©(Ÿ9P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚18699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148272896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173196715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173196715 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173206692 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚645964825315840 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2827583355670528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19776436148224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2827583338631168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32753298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40189440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19776436148224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2847329960325206 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19746604654678 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2842203143979008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19776777487360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72942738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861979938505728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19991214659072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19746264104788 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚16015.837596989 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72943015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861979580127232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1539 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321479590 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148272898 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645964934602240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2847329602735956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625973701330944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148272896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843563571438782e+09_f;Õ ( 9†P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861979580127232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1539 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321479590 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148272898 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645964934602240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2847329602735956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625973701330944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148272896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843563571438782e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚18699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148272896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173196715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173196715 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173206692 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚645964825315840 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2827583355670528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19776436148224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2827583338631168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32753298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40189440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19776436148224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2847329960325206 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19746604654678 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2842203143979008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19776777487360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72942738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861979938505728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19991214659072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19746264104788 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚16015.837596989 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72943015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ƒ RäG4¡9/ _wandbruntime‚16016  _runtime‚16016í©R ¢9ŠN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2827729849303040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2847477482050912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2827729866342400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2862127833783808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚645999379551232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19992135988224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173215758 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72946578 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16016.451069539 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148280530 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321496288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173205767 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40191802 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321486295 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173205767 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19777448775168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40191535 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19747615708512 trainer/global_step‚18699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32755043 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626007130089472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626007148701696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2847477124461550 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2842350402048000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321486295 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173215758 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645999266077696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19777448775168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976%½±…'£9N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2827729849303040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2847477482050912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2827729866342400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2862127833783808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚645999379551232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19992135988224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173215758 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72946578 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16016.451069539 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148280530 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321496288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173205767 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40191802 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321486295 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173205767 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19777448775168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40191535 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19747615708512 trainer/global_step‚18699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32755043 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626007130089472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626007148701696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2847477124461550 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2842350402048000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321486295 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173215758 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645999266077696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19777448775168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0„ HcÃ4¤9/ _wandbruntime‚16016  _runtime‚16016Å„ÛZ¥9Œ /loss/FRt2/detection_loss‚1.1490631400808926 0loss/FBt2/detection_loss‚0.41615126564986654 $ _timestamp‚1.784356357760897e+09 _runtime‚16016.454215117 epoch‚46 0loss/FRt1/assignment_loss‚1.3100564017022918 3loss/SRqqt2/assignment_loss‚0.44946378801117676 4loss/FBt1/assignment_loss‚0.00013525064650699343 /loss/FRt1/detection_loss‚0.5691469358678269 1loss/SRqqt1/detection_loss‚0.7445376316114912 0loss/FRt2/assignment_loss‚0.7270812308597734 3loss/FBt2/assignment_loss‚1.324670593886098e-06 _step‚1541 0loss/SRqqt2/detection_loss‚0.538949677485044 /loss/FBt1/detection_loss‚0.5251731392520014 &loss/total_loss‚6.8267153391301445 trainer/global_step‚18699 3loss/SRqqt1/assignment_loss‚0.39695555329267995‰ðnª¦9‘ 0loss/FBt2/detection_loss‚0.41615126564986654 $ _timestamp‚1.784356357760897e+09 _runtime‚16016.454215117 epoch‚46 0loss/FRt1/assignment_loss‚1.3100564017022918 3loss/SRqqt2/assignment_loss‚0.44946378801117676 4loss/FBt1/assignment_loss‚0.00013525064650699343 /loss/FRt1/detection_loss‚0.5691469358678269 1loss/SRqqt1/detection_loss‚0.7445376316114912 0loss/FRt2/assignment_loss‚0.7270812308597734 3loss/FBt2/assignment_loss‚1.324670593886098e-06 _step‚1541 0loss/SRqqt2/detection_loss‚0.538949677485044 /loss/FBt1/detection_loss‚0.5251731392520014 &loss/total_loss‚6.8267153391301445 trainer/global_step‚18699 3loss/SRqqt1/assignment_loss‚0.39695555329267995 /loss/FRt2/detection_loss‚1.1490631400808926… ºôáý“§9‚(:ˆ ‰¼ìÒ’ƒŒ• gpu.0.gpu‚52.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F'§Ñ/¨9‚(:¤ ‰¼ìÒû network.sent‚ 3.0020483e+07$ disk./.usageGB‚585.6297836303711 disk.rbd1.out‚ 49.9736328125 disk.dm-1.in‚ 233.44140625 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 27293.2734375 network.recv‚ 4.258614e+06 disk.dm-0.in‚27279.23828125 disk.dm-0.out‚35161.76171875 disk.sda2.in‚ 82.544921875* disk./.usagePercent‚39.302106155551655 disk.sda1.out‚0* proc.memory.percent‚17.109203338623047 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.drbd2165.out‚0 disk.sda.in‚541.0478515625 disk.sda.out‚ 1712.40234375( proc.memory.availableMB‚ 373.35546875 disk.dm-1.out‚ 1541.8671875! proc.memory.rssMB‚ 2803.171875 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1541.8671875 disk.sda1.in‚ 8.759765625% disk.nvme0c0n1.out‚35161.76171875 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 27293.2734375# disk.nvme0n1.out‚35161.76171875$ memory_percent‚97.72121906280518 disk.rbd0.out‚0 cpu‚26.167311462915645 proc.cpu.threads‚17Кwÿ‘©9‚(:† ˜¼ìÒÚż“ gpu.0.gpu‚41.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Ĺ(ª9‚(: ˜¼ìÒÒµë•" disk.nvme0n1.in‚27397.63671875# memory_percent‚97.7224349975586 disk.dm-1.in‚ 233.44140625$ disk.nvme0c0n1.in‚27397.63671875% disk.nvme0c0n1.out‚35164.65234375 disk.drbd2165.out‚0) disk./.usagePercent‚39.30210282745371 disk.sda1.out‚0 disk.sda3.out‚ 1542.296875! proc.memory.rssMB‚ 2803.171875 disk.rbd1.out‚ 49.9736328125 disk.sda.in‚541.0478515625 disk.sda.out‚ 1712.83203125 disk.sda2.in‚ 82.544921875% proc.memory.availableMB‚ 373.15625 cpu‚26.16686035279532 disk.rbd1.in‚ 2.1796875 disk.dm-1.out‚ 1542.296875 disk.sda2.out‚ 170.53515625* proc.memory.percent‚17.109203338623047 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.dm-0.out‚35164.65234375 disk.sda3.in‚ 417.107421875 proc.cpu.threads‚17 network.sent‚ 3.0083009e+07 network.recv‚ 4.262765e+06 disk.drbd2165.in‚ 2.25390625# disk.nvme0n1.out‚35164.65234375$ disk./.usageGB‚585.6297340393066 disk.dm-0.in‚ 27383.6015625 disk.rbd0.in‚ 0.16015625.¦ž«9‚(:“ §¼ìÒÞ·À“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mLôc:¬9‚(:¯ §¼ì҇º–$ disk./.usageGB‚585.6297798156738 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0& disk.nvme0n1.out‚35170.39013671875* proc.memory.percent‚17.109203338623047 disk.sda3.in‚ 417.107421875$ memory_percent‚97.72121906280518 disk.dm-1.out‚ 1542.73828125 network.sent‚ 3.0088625e+07( disk.nvme0c0n1.out‚35170.39013671875 disk.dm-0.in‚ 27383.6015625 disk.sda2.out‚ 170.53515625 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625 network.recv‚ 4.266289e+06 disk.rbd1.out‚ 50.0361328125 disk.sda1.in‚ 8.759765625 disk.sda3.out‚ 1542.73828125! proc.memory.rssMB‚ 2803.171875 disk.sda1.out‚0# disk.dm-0.out‚35170.39013671875 cpu‚26.166453786684393( proc.memory.availableMB‚ 373.35546875* disk./.usagePercent‚39.302105899544124 disk.sda.out‚ 1713.2734375 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875$ disk.nvme0c0n1.in‚27397.63671875 disk.sda.in‚541.0478515625" disk.nvme0n1.in‚27397.63671875[–eä4­9/ _wandbruntime‚16055  _runtime‚16055hj%$Ê®9Ä & lr-AdamW/pg1‚4.498910550950323e-06 & lr-AdamW/pg2‚4.498910550950323e-06 trainer/global_step‚18749 % _timestamp‚1.7843563972175703e+09 _runtime‚16055.911119973 _step‚1542†:z¨Ï¯9É trainer/global_step‚18749 % _timestamp‚1.7843563972175703e+09 _runtime‚16055.911119973 _step‚1542 & lr-AdamW/pg1‚4.498910550950323e-06 & lr-AdamW/pg2‚4.498910550950323e-06† %ôB4°9/ _wandbruntime‚16055  _runtime‚16055yÔCR $±9€P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2849565212499968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2849565195460608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20042552652800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322305663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2834908872224768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322315642 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚18749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73134990 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40294179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32840548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2849565195460608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2834908889264128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚1543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2869392685489152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19827131650048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19827472989184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148654498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173661144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32840551 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148654496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73134727 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19796880591696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19827472989184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647687611721728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2849565212499968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19797221141586 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚627645157869568 _runtime‚16055.912466284 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2869392327110656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19827131650048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148654498 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚647687710522368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7843563972187552e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40294439 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2854706110405714 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2869392685489152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2869392327110656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20042472464384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647687611721728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2849565212499968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19797221141586 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚627645157869568 _runtime‚16055.912466284 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2869392327110656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19827131650048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148654498 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚647687710522368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7843563972187552e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40294439 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2854706110405714 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2869392685489152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2869392327110656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20042472464384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2834908872224768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322315642 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚18749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73134990 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40294179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32840548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2849565195460608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2834908889264128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚1543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2869392685489152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19827131650048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19827472989184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148654498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173661144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32840551 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148654496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438‡ äÑ%4³9/ _wandbruntime‚16056  _runtime‚16056)¦ÀÞ'´9‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2849712453529600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2869540939152896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40296545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73138576 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173660219 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173670210 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2849712436490240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2854853274542058 _step‚1544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19828485623296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚18749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2869540580767232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20043492609536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2849712453529600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚627678568015872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148662130 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2869540939152896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322332340 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2849712436490240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20043579082240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148662128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32842293 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322332340 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647722165710336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2835055399936000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19797891645418 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19828144276992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322322347 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19828144276992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843563978339422e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148662128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73138841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E¼½T'µ9N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚18749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2869540580767232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20043492609536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2849712453529600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚627678568015872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148662130 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2869540939152896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322332340 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2849712436490240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20043579082240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148662128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32842293 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322332340 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647722165710336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2835055399936000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19797891645418 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19828144276992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322322347 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19828144276992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843563978339422e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148662128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73138841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73138576 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173660219 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173670210 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2849712436490240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2854853274542058 _step‚1544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19828485623296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ˆ ÐÕöq4¶9/ _wandbruntime‚16056  _runtime‚16056òœ6z ·9‡ 0loss/FRt2/assignment_loss‚0.7257649787636691 3loss/SRqqt2/assignment_loss‚0.39956345330806303 /loss/FRt1/detection_loss‚0.5951000686540405 _runtime‚16056.52943437 _step‚1545 4loss/FBt1/assignment_loss‚1.9073466631880364e-07 2loss/FBt2/assignment_loss‚9.29336777276084e-05 .loss/FRt2/detection_loss‚1.203909110395096 1loss/SRqqt2/detection_loss‚0.5478953453393818 /loss/FBt1/detection_loss‚0.6039669936383143 %loss/total_loss‚7.166283154305968 $ _timestamp‚1.784356397836115e+09 0loss/FRt1/assignment_loss‚1.5100641755418405 2loss/SRqqt1/assignment_loss‚0.3998391573463788 /loss/FBt2/detection_loss‚0.4138006787565688 trainer/global_step‚18749 1loss/SRqqt1/detection_loss‚0.7662860681502212 epoch‚46Wã '¸9Œ /loss/FBt2/detection_loss‚0.4138006787565688 trainer/global_step‚18749 1loss/SRqqt1/detection_loss‚0.7662860681502212 epoch‚46 0loss/FRt2/assignment_loss‚0.7257649787636691 3loss/SRqqt2/assignment_loss‚0.39956345330806303 /loss/FRt1/detection_loss‚0.5951000686540405 _runtime‚16056.52943437 _step‚1545 4loss/FBt1/assignment_loss‚1.9073466631880364e-07 2loss/FBt2/assignment_loss‚9.29336777276084e-05 .loss/FRt2/detection_loss‚1.203909110395096 1loss/SRqqt2/detection_loss‚0.54789534533938êCãë18 /loss/FBt1/detection_loss‚0.6039669936383143 %loss/total_loss‚7.166283154305968 $ _timestamp‚1.784356397836115e+09 0loss/FRt1/assignment_loss‚1.5100641755418405 2loss/SRqqt1/assignment_loss‚0.3998391573463788‰ Š ŸJž¹9‚(:“ ¶¼ìÒíþ “ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.464800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03h…|0º9‚(:¥ ¶¼ìÒ”ÓÔ– disk.rbd1.out‚ 50.0361328125 proc.cpu.threads‚17) disk./.usagePercent‚39.30211793189823$ disk.nvme0c0n1.in‚27397.63671875 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625$ memory_percent‚97.72281646728516* proc.memory.percent‚17.109203338623047( disk.nvme0c0n1.out‚35173.36376953125 disk.sda3.in‚ 417.107421875 disk.drbd2165.in‚ 2.25390625 disk.sda3.out‚ 1543.09375 disk.rbd1.in‚ 2.1796875 disk.dm-1.in‚ 233.44140625 cpu‚26.166033933126574 network.recv‚ 4.269583e+06# disk.dm-0.out‚35173.36376953125 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0" disk.nvme0n1.in‚27397.63671875& disk.nvme0n1.out‚35173.36376953125 disk.dm-1.out‚ 1543.09375! proc.memory.rssMB‚ 2803.171875$ disk./.usageGB‚585.6299591064453 disk.drbd2165.out‚0 disk.dm-0.in‚ 27383.6015625 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875 disk.sda1.out‚0% proc.memory.availableMB‚ 373.09375 network.sent‚ 3.009367e+07 disk.sda.out‚ 1713.62890625;@š»9‚(: żìҨݶ• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚243.5' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.699999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a>~7¼9‚(:¬ żìÒÞ‚ì– disk.sda.in‚541.0478515625 disk.sda.out‚ 1713.93359375 disk.dm-1.in‚ 233.44140625$ disk.nvme0c0n1.in‚27397.63671875( proc.memory.availableMB‚ 373.11328125 cpu‚26.165614876432443 proc.cpu.threads‚17 disk.dm-0.in‚ 27383.6015625 disk.sda2.in‚ 82.544921875 network.recv‚ 4.274078e+06 disk.sda2.out‚ 170.53515625 disk.sda1.out‚0 disk.drbd2165.out‚0# memory_percent‚97.7226972579956* proc.memory.percent‚17.109203338623047# disk.dm-0.out‚35179.22607421875 disk.rbd0.in‚ 0.16015625! proc.memory.rssMB‚ 2803.171875$ disk./.usageGB‚585.6299057006836" disk.nvme0n1.in‚27397.63671875& disk.nvme0n1.out‚35179.22607421875 disk.drbd2165.in‚ 2.25390625 network.sent‚ 3.0156976e+07) disk./.usagePercent‚39.30211434779275 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1543.3984375 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 50.0986328125 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1543.3984375( disk.nvme0c0n1.out‚35179.22607421875.vצ“½9‚(:ˆ Ô¼ìÒ©µ±“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d÷Ôß(¾9‚(: Ô¼ìÒ–‡¡– network.sent‚ 3.0161485e+07 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.sda2.in‚ 82.544921875$ memory_percent‚97.72202968597412 disk.drbd2165.out‚0 disk.sda.out‚ 1714.36328125 disk.nvme0n1.in‚ 27511.359375 disk.sda3.in‚ 417.107421875* proc.memory.percent‚17.109203338623047 disk.dm-1.out‚ 1543.828125$ disk./.usageGB‚585.6298561096191 disk.rbd0.in‚ 0.16015625 disk.sda3.out‚ 1543.828125 proc.cpu.threads‚17 disk.dm-1.in‚ 233.44140625 disk.dm-0.in‚27497.32421875 disk.rbd1.in‚ 2.1796875 network.recv‚ 4.277213e+06( disk./.usagePercent‚39.3021110196948 disk.sda.in‚541.0478515625 disk.drbd2165.in‚ 2.25390625 disk.rbd1.out‚ 50.0986328125 cpu‚26.16516814918584( proc.memory.availableMB‚ 373.22265625" disk.nvme0c0n1.in‚ 27511.359375 disk.rbd0.out‚0 disk.sda2.out‚ 170.53515625# disk.nvme0n1.out‚35182.31640625! proc.memory.rssMB‚ 2803.171875% disk.nvme0c0n1.out‚35182.31640625 disk.dm-0.out‚35182.31640625ËŒÊ÷4¿9/ _wandbruntime‚16095  _runtime‚16095b@ª^ÊÀ9Ä & lr-AdamW/pg1‚4.147561372452325e-06 & lr-AdamW/pg2‚4.147561372452325e-06 trainer/global_step‚18799 % _timestamp‚1.7843564370929363e+09 _runtime‚16095.786500441 _step‚1546-]ÍÁÏÁ9É & lr-AdamW/pg2‚4.147561372452325e-06 trainer/global_step‚18799 % _timestamp‚1.7843564370929363e+09 _runtime‚16095.786500441 _step‚1546 & lr-AdamW/pg1‚4.147561372452325e-06Š ª;c-4Â9/ _wandbruntime‚16095  _runtime‚16095Ìõ™é(Ã9€P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73326390 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2856927246942208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19878168491008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40398862 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629316577183744 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2842234405818368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174105619 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19877827151872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629316595795968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2862082260486222 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19877827151872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174115596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2842234422857728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2876805074094080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20093912818688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 trainer/global_step‚18799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19878168491008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2862081902896972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149036098 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149036096 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2856927246942208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20093824241664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19847837628494 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174115596 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323141715 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174105619 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876805432472576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16095.787728188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32927798 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2856927263981568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323151694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149036098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2876805432472576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚649410401425408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2856927263981568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32927801 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19847497078604 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40398592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1547 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚649410508614656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876805074094080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149036096 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876805074094080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149036096 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323141715 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174105619 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876805432472576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16095.787728188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32927798 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2856927263981568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323151694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149036098 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2876805432472576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚649410401425408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2856927263981568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32927801 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19847497078604 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40398592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1547 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚649410508614656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0‹ É8_Ô4Å9/ _wandbruntime‚16096  _runtime‚16096µ(?û Æ9ŠN YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19879181125120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20094942504448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20094868614656 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2857074487971840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174124662 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2857074505011200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649444874556928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174114671 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149043728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2876953686136320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19848848682328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16096.401529804 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323158399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32929546 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2862229424622566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚73889792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843564377077634e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73330251 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629350005942272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73330495 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323168392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚92502016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174114671 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323158399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1548 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149043728Í5uk'Ç9N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚73889792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843564377077634e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73330251 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629350005942272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73330495 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323168392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚92502016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174114671 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323158399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1548 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149043728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19879181125120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20094942504448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20094868614656 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2857074487971840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174124662 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2857074505011200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649444874556928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174114671 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149043728 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2876953686136320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19848848682328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16096.401529804 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323158399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32929546 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2862229424622566Œ IŠ:g4È9/ _wandbruntime‚16096  _runtime‚16096ƒS\†É9Œ 2loss/SRqqt1/assignment_loss‚0.4446083349904527 4loss/FBt1/assignment_loss‚0.00037699036715443945 2loss/SRqqt2/detection_loss‚0.49813673131573316 0loss/FBt1/detection_loss‚0.17290051688723906 epoch‚46 _runtime‚16096.404599196 _step‚1549 2loss/SRqqt2/assignment_loss‚0.4276503929217536 /loss/FRt2/detection_loss‚1.1704317807871538 1loss/SRqqt1/detection_loss‚0.8337023053444317 /loss/FBt2/detection_loss‚0.5744050472549636 trainer/global_step‚18799 % _timestamp‚1.7843564377112882e+09 0loss/FRt1/assignment_loss‚1.5254203193049574 0loss/FRt2/assignment_loss‚0.7947280259077786 3loss/FBt2/assignment_loss‚6.794681419052934e-05 /loss/FRt1/detection_loss‚0.4922247585555781 %loss/total_loss‚6.934653150451387¢‹éÊ9‘ epoch‚46 _runtime‚16096.404599196 _step‚1549 2loss/SRqqt2/assignment_loss‚0.4276503929217536 /loss/FRt2/detection_loss‚1.1704317807871538 1loss/SRqqt1/detection_loss‚0.8337023053444317 /loss/FBt2/detection_loss‚0.5744050472549636 trainer/global_step‚18799 % _timestamp‚1.7843564377112882e+09 0loss/FRt1/assignment_loss‚1.5254203193049574 0loss/FRt2/assignment_loss‚0.7947280259077786 3loss/FBt2/assignment_loss‚6.794681419052934e-05 /loss/FRt1/detection_loss‚0.4922247585555781 %loss/total_loss‚6.934653150451387 2loss/SRqqt1/assignment_loss‚0.4446083349904527 4loss/FBt1/assignment_loss‚0.00037699036715443945 2loss/SRqqt2/detection_loss‚0.49813673131573316 0loss/FBt1/detection_loss‚0.17290051688723906 DèéœË9‚(:‘ ã¼ìÒ˜ “– gpu.0.gpu‚93.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>BG ,Ì9‚(:¡ ã¼ìÒÌóÔ— disk.rbd0.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚35189.826171875# memory_percent‚97.7405309677124) proc.memory.percent‚17.12520122528076 network.sent‚ 3.0167113e+07 network.recv‚ 4.280689e+06! disk.dm-0.out‚35189.826171875( proc.memory.availableMB‚ 370.19140625 disk.sda.in‚541.0478515625 disk.sda.out‚ 1714.64453125 disk.sda2.out‚ 170.53515625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1544.109375" disk.nvme0c0n1.in‚ 27511.359375 disk.dm-0.in‚27497.32421875) disk./.usagePercent‚39.30211229973247 disk.nvme0n1.in‚ 27511.359375$ disk.nvme0n1.out‚35189.826171875 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625 cpu‚26.16793763445894 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625 disk.sda3.out‚ 1544.109375 disk.sda2.in‚ 82.544921875# proc.memory.rssMB‚ 2805.79296875 proc.cpu.threads‚17$ disk./.usageGB‚585.6298751831055 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 50.1611328125Uä%4Í9/ _wandbruntime‚16113  _runtime‚16113ÄK‹=Î9…{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.56636643409729 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24673068523406982 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12443544715642929 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 particle/accuracy_1_of_3‚1 trainer/global_step‚18799 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24451152980327606 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7242573499679565 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6125900745391846 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739060640335083 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6333988904953003 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025547944009304047 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18628065288066864 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2816808819770813 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.723237156867981 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24763935804367065 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6259750127792358 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7669652104377747 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035989366471767426 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28170499205589294 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6556813716888428 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025320276618003845 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3374963700771332 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_2_of_2‚0.12624725699424744 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8219576478004456 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23504102230072021 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 +jet/accuracy_1_of_1‚0.34618881344795227 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7934552431106567 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23018264770507812 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9806751012802124 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.668467104434967 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28140968084335327 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7084254026412964 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9431183934211731 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.853363573551178 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.638804018497467 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7322563529014587 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8763743042945862 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 *jet/accuracy_2_of_3‚0.5102413892745972 +jet/accuracy_3_of_3‚0.03792091831564903 % _timestamp‚1.7843564550322266e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6919456720352173 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5330705642700195 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23995338380336761 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7101496458053589 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6377795338630676 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5684841275215149 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27324214577674866 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23676490783691406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2816808819770813 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1345292180776596 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.37918663024902344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09553547203540802 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.8981678485870361 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22111542522907257 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9262031316757202 'particle/f_score‚0.6050757169723511 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 )loss/val_total_loss‚7.715428183547539 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6377795338630676 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.938788890838623 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2948853671550751 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3144916296005249 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09651567786931992 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14835700392723083 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7148341536521912 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6576691269874573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 (particle/accuracy‚0.8331279754638672 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5489365458488464 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.727645218372345 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7232769727706909 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03620557114481926 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32823672890663147 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3636297881603241 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37669312953948975 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6845842599868774 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025629499927163124 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16525480151176453 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0854882299900055 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.03814428672194481 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27768003940582275 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7643120884895325 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025446776300668716 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6548420786857605 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04432744160294533 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21211913228034973 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _runtime‚16113.726500305 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.653657853603363 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6846463084220886 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21145647764205933 particle/accuracy_3_of_5‚1 _step‚1550 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7737412452697754 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6697142720222473 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.785712480545044 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23686639964580536 particle/accuracy_2_of_4‚1 *particle/sensitivity‚0.595304548740387 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7279372215270996 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6908407211303711 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29237374663352966 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6282214522361755 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7948760986328125 epoch‚46 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7843247652053833 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.791155219078064 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7855436205863953 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2480023205280304 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2921234369277954 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15118509531021118 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1428944319486618 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049426738172769547 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_2‚1Gµ!½Ï9Š{ @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21211913228034973 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.01155598927289ôÎéÓ:2475 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _runtime‚16113.726500305 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.653657853603363 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6846463084220886 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21145647764205933 particle/accuracy_3_of_5‚1 _step‚1550 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7737412452697754 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6697142720222473 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.785712480545044 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23686639964580536 particle/accuracy_2_of_4‚1 *particle/sensitivity‚0.595304548740387 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7279372215270996 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6908407211303711 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29237374663352966 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6282214522361755 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7948760986328125 epoch‚46 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7843247652053833 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.791155219078064 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7855436205863953 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2480023205280304 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2921234369277954 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15118509531021118 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1428944319486618 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049426738172769547 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.56636643409729 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24673068523406982 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12443544715642929 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 particle/accuracy_1_of_3‚1 trainer/global_step‚18799 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24451152980327606 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7242573499679565 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6125900745391846 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739060640335083 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6333988904953003 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025547944009304047 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18628065288066864 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2816808819770813 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.723237156867981 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24763935804367065 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6259750127792358 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7669652104377747 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.035989366471767426 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28170499205589294 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6556813716888428 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025320276618003845 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3374963700771332 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_2_of_2‚0.12624725699424744 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8219576478004456 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23504102230072021 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 +jet/accuracy_1_of_1‚0.34618881344795227 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7934552431106567 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23018264770507812 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9806751012802124 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.668467104434967 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28140968084335327 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7084254026412964 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9431183934211731 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.853363573551178 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.638804018497467 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7322563529014587 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8763743042945862 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 *jet/accuracy_2_of_3‚0.5102413892745972 +jet/accuracy_3_of_3‚0.03792091831564903 % _timestamp‚1.7843564550322266e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6919456720352173 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5330705642700195 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23995338380336761 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7101496458053589 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6377795338630676 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5684841275215149 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27324214577674866 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23676490783691406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2816808819770813 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1345292180776596 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.37918663024902344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09553547203540802 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.8981678485870361 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22111542522907257 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9262031316757202 'particle/f_score‚0.6050757169723511 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 )loss/val_total_loss‚7.715428183547539 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6377795338630676 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.938788890838623 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2948853671550751 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3144916296005249 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09651567786931992 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14835700392723083 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7148341536521912 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6576691269874573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 (particle/accuracy‚0.8331279754638672 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5489365458488464 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.727645218372345 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7232769727706909 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03620557114481926 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32823672890663147 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3636297881603241 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37669312953948975 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6845842599868774 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025629499927163124 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16525480151176453 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0854882299900055 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.03814428672194481 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27768003940582275 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7643120884895325 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025446776300668716 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6548420786857605 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04432744160294533Ž 9‘y“Ð9‚(:ˆ ò¼ìҸ˸“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÁã6Ñ9‚(:« ò¼ìÒÃÀ— disk.dm-1.out‚ 1544.30078125 disk.drbd2165.in‚ 2.25390625# disk./.usageGB‚585.629940032959 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚35193.61767578125 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚35193.61767578125 disk.sda3.in‚ 417.107421875 disk.sda1.out‚0 disk.dm-0.in‚27497.32421875 disk.rbd1.in‚ 2.1796875# proc.memory.rssMB‚ 2806.44921875) disk./.usagePercent‚39.30211665186056 disk.sda3.out‚ 1544.30078125 network.recv‚ 4.284903e+06 disk.rbd0.out‚0 cpu‚26.168526797551863 disk.rbd0.in‚ 0.16015625$ memory_percent‚97.74374961853027* proc.memory.percent‚17.129206657409668 proc.cpu.threads‚17 disk.sda.out‚ 1714.8359375 disk.nvme0n1.in‚ 27511.359375 disk.rbd1.out‚ 50.1611328125' proc.memory.availableMB‚ 369.6640625 disk.drbd2165.out‚0 disk.sda2.in‚ 82.544921875" disk.nvme0c0n1.in‚ 27511.359375( disk.nvme0c0n1.out‚35193.61767578125 network.sent‚ 3.0229629e+07 disk.dm-1.in‚ 233.44140625`‹òª“Ò9‚(:ˆ ½ìÒ‘Á¬“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Ä’$/Ó9‚(:¤ ½ìÒ­Û¯–$ disk./.usageGB‚585.6300010681152 disk.rbd1.out‚ 50.2236328125 disk.dm-1.out‚ 1544.73046875" disk.nvme0c0n1.in‚ 27511.359375) disk./.usagePercent‚39.30212074798111 disk.sda1.out‚0* proc.memory.percent‚17.129206657409668 disk.drbd2165.out‚0 network.recv‚ 4.289308e+06 disk.dm-0.in‚27497.32421875 proc.cpu.threads‚17 network.sent‚ 3.0287126e+07 disk.sda3.out‚ 1544.73046875 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.sda.in‚541.0478515625 disk.sda.out‚ 1715.265625& proc.memory.availableMB‚ 369.328125 disk.sda1.in‚ 8.759765625& disk.nvme0c0n1.out‚35198.798828125! disk.dm-0.out‚35198.798828125 disk.sda2.in‚ 82.544921875# proc.memory.rssMB‚ 2806.44921875 cpu‚26.168109710785853 disk.nvme0n1.in‚ 27511.359375$ disk.nvme0n1.out‚35198.798828125 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.dm-1.in‚ 233.44140625 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625$ memory_percent‚97.74580001831055äÃ:%4Ô9/ _wandbruntime‚16153  _runtime‚16153éÅ£JÊÕ9Ä & lr-AdamW/pg2‚3.810367490266805e-06 trainer/global_step‚18849 % _timestamp‚1.7843564945282378e+09 _runtime‚16153.221790588 _step‚1551 & lr-AdamW/pg1‚3.810367490266805e-069TF–ÏÖ9É & lr-AdamW/pg1‚3.810367490266805e-06 & lr-AdamW/pg2‚3.810367490266805e-06 trainer/global_step‚18849 % _timestamp‚1.7843564945282378e+09 _runtime‚16153.221790588 _step‚1551 Ñ 4×9/ _wandbruntime‚16153  _runtime‚16153£/ë (Ø9P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19946255859200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19946597198336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40533468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚651167086061568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19915833290741 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843564945293236e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2866218865074176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149538364 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324178734 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33017571 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2871384435916789 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚16153.223017243 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174640370 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149538362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20163158851584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33017568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚631003927209984 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2871384793506039 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19916173840631 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2886165120933376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174630393 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2886165479311872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2851468619665408 _step‚1552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174630393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631003945822208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19946597198336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20163241137152 trainer/global_step‚18849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2886165120933376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324168755 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2851468602626048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2866218865074176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2886165479311872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149538362 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19946255859200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174640370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40533720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2866218882113536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ÄíânMÙ9†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149538364 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324178734 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33017571 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2871384435916789 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚16153.223017243 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174640370 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149538362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20163158851584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33017568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚631003927209984 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2871384793506039 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19916173840631 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2886165120933376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174630393 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2886165479311872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2851468619665408 _step‚1552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/nÏ@&§¿um_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174630393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631003945822208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19946597198336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20163241137152 trainer/global_step‚18849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2886165120933376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324168755 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2851468602626048 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2866218865074176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2886165479311872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149538362 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19946255859200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174640370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40533720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2866218882113536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19946255859200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19946597198336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40533468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚651167086061568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19915833290741 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843564945293236e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2866218865074176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cm4Ú9/ _wandbruntime‚16153  _runtime‚16153kûó'Û9‹N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149545996 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20164192948736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2851615113297920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33019316 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324185439 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324185439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2851615130337280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20164275227136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174649436 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40535799 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631037374580736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚16153.836356881 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2886313374589952 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2866366123143168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149545996 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174639445 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19947268486144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73554867 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73555115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324195432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651201649807872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2866366106103808 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19947609832448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631037355968512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19947268486144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174639445 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2871531957642383 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2866366123143168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18849 % _timestamp‚1.7843564951425068e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2866366106103808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40535799 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631037374580736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚16153.836356881 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2886313374589952 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2866366123143168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149545996 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174639445 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19947268486144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73554867 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73555115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324195432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651201649807872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2866366106103808 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19947609832448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631037355968512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19947268486144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174639445 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2871531957642383 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2866366123143168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18849 % _timestamp‚1.7843564951425068e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2866366106103808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20164192948736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2851615113297920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33019316 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324185439 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324185439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2851615130337280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20164275227136‘ ª³F4Ý9/ _wandbruntime‚16153  _runtime‚16153¢¬xQÞ9‰ 2loss/SRqqt2/assignment_loss‚0.4671983613621426 1loss/FBt2/assignment_loss‚0.01182464149461922 /loss/FRt1/detection_loss‚0.5790806488704962 0loss/FBt2/detection_loss‚0.29074708558888857 1loss/SRqqt2/detection_loss‚0.5954755213166613 %loss/total_loss‚6.939247119968659 trainer/global_step‚18849 _step‚1554 0loss/FRt2/assignment_loss‚0.7743514002885837 3loss/FBt1/assignment_loss‚0.0002387026181757963 1loss/SRqqt1/detection_loss‚0.8366331053353152 0loss/FBt1/detection_loss‚0.42658398893949995 epoch‚47 _runtime‚16153.838906152 /loss/FRt1/assignment_loss‚1.418946444655264 3loss/SRqqt1/assignment_loss‚0.35824884787120664 /loss/FRt2/detection_loss‚1.1799183716278052 % _timestamp‚1.7843564951456006e+09oˆ Fß9Ž 3loss/SRqqt1/assignment_loss‚0.35824884787120664 /loss/FRt2/detection_loss‚1.1799183716278052 % _timestamp‚1.7843564951456006e+09 2loss/SRqqt2/assignment_loss‚0.4671983613621426 1loss/FBt2/assignment_loss‚0.01182464149461922 /loss/FRt1/detection_loss‚0.5790806488704962 0loss/FBt2/detection_loss‚0.29074708558888857 1loss/SRqqt2/detection_loss‚0.5954755213166613 %loss/total_loss‚6.939247119968659 trainer/global_step‚18849 _step‚1554 0loss/FRt2/assignment_loss‚0.7743514002885837 3loss/FBt1/assignment_loss‚0.0002387026181757963 1loss/SRqqt1/detection_loss‚0.8366331053353152 0loss/FBt1/detection_loss‚0.42658398893949995 epoch‚47 _runtime‚16153.838906152 /loss/FRt1/assignment_loss‚1.418946444655264’ 'žx­“à9‚(:ˆ ½ìÒ¯–³” gpu.0.gpu‚55.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰âÈÏ4á9‚(:© ½ìÒþ›¶—$ disk./.usageGB‚585.6300010681152 disk.sda.out‚ 1715.68359375 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1545.1484375 disk.drbd2165.out‚0) disk./.usagePercent‚39.30212074798111& disk.nvme0n1.out‚35202.41259765625 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625( disk.nvme0c0n1.out‚35202.41259765625# proc.memory.rssMB‚ 2806.44921875 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625$ memory_percent‚97.74723052978516 proc.cpu.threads‚17 disk.rbd1.out‚ 50.2236328125% proc.memory.availableMB‚ 369.09375* proc.memory.percent‚17.129206657409668 disk.dm-0.in‚27506.73046875 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0 cpu‚26.167687768295092 network.sent‚ 3.0292705e+07 network.recv‚ 4.292537e+06 disk.nvme0n1.in‚ 27520.765625 disk.sda3.out‚ 1545.1484375# disk.dm-0.out‚35202.41259765625 disk.sda.in‚541.0478515625" disk.nvme0c0n1.in‚ 27520.765625磾L“â9‚(:ˆ Ÿ½ìÒ×ý½” gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vï9ã9‚(:® Ÿ½ìÒ¥«õ• disk.dm-0.in‚ 27618.9921875# disk.dm-0.out‚35207.37158203125 disk.rbd1.out‚ 50.3486328125& proc.memory.availableMB‚ 368.890625 network.sent‚ 3.0355283e+07 disk.dm-1.out‚ 1545.70703125 disk.sda.in‚541.0478515625" disk.nvme0n1.in‚27633.02734375 disk.drbd2165.out‚0 disk.sda.out‚ 1716.2421875 disk.sda2.in‚ 82.544921875 proc.cpu.threads‚17 network.recv‚ 4.297096e+06$ disk./.usageGB‚585.6288261413574 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚27633.02734375& disk.nvme0n1.out‚35207.37158203125 disk.sda3.in‚ 417.107421875( disk.nvme0c0n1.out‚35207.37158203125 disk.rbd1.in‚ 2.1796875 disk.sda1.in‚ 8.759765625* proc.memory.percent‚17.129206657409668$ memory_percent‚97.74847030639648 disk.sda3.out‚ 1545.70703125 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0# proc.memory.rssMB‚ 2806.44921875 cpu‚26.167259723852624) disk./.usagePercent‚39.30204189766056 disk.drbd2165.in‚ 2.25390625õìô“ä9‚(:ˆ ®½ìÒ缩“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ß}Œ6å9‚(:« ®½ìÒÿ¾´•$ disk.nvme0c0n1.in‚27633.02734375 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚585.6288948059082" disk.dm-0.out‚35210.6748046875 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1546.2109375 disk.rbd1.out‚ 50.3486328125 cpu‚26.166844577176054 disk.drbd2165.out‚0 disk.dm-0.in‚ 27618.9921875 disk.rbd1.in‚ 2.1796875 disk.dm-1.out‚ 1546.2109375 disk.sda.out‚ 1716.74609375 disk.drbd2165.in‚ 2.25390625 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27633.02734375 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0* proc.memory.percent‚17.129206657409668$ memory_percent‚97.74563312530518( proc.memory.availableMB‚ 369.35546875 network.recv‚ 4.300589e+06% disk.nvme0n1.out‚35210.6748046875) disk./.usagePercent‚39.30204650579617 disk.sda.in‚541.0478515625 network.sent‚ 3.036043e+07 disk.dm-1.in‚ 233.44140625' disk.nvme0c0n1.out‚35210.6748046875# proc.memory.rssMB‚ 2806.44921875‚Ú}4æ9/ _wandbruntime‚16193  _runtime‚16193zMF–ç9Ä & lr-AdamW/pg2‚3.487350561726715e-06 trainer/global_step‚18899 % _timestamp‚1.7843565344476275e+09 _runtime‚16193.141276025 tÜË4_step‚1555 & lr-AdamW/pg1‚3.487350561726715e-06)ÖÑØÏè9É & lr-AdamW/pg1‚3.487350561726715e-06 & lr-AdamW/pg2‚3.487350561726715e-06 trainer/global_step‚18899 % _timestamp‚1.7843565344476275e+09 _runtime‚16193.141276025 _step‚1555“ e‰á4é9/ _wandbruntime‚16193  _runtime‚16193k7[(ê9P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33104821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2858794136219648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149919962 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2893578226295296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19966790327539 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2873580916555776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1556 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16193.142607929 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325014786 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175094822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40637677 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19996951361024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73742498 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149919962 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175084845 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149919964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19997292700160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 % _timestamp‚1.7843565344488823e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652890016345600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40637417 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2878760585997297 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2873580916555776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73742235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632675383748608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2893577867916800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325004807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2878760943586547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632675365136384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2873580933595136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚18899 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19997292700160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20214546117120 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2893578226295296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2858794153259008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2873580933595136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149919964 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175094822 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20214632596992«… (ë9†P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2873580933595136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚18899 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19997292700160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20214546117120 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2893578226295296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2858794153259008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2873580933595136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149919964 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175094822 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20214632596992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33104821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2858794136219648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149919962 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2893578226295296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19966790327539 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2873580916555776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1556 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16193.142607929 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325014786 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175094822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40637677 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19996951361024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73742498 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149919962 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175084845 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149919964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19997292700160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 % _timestamp‚1.7843565344488823e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652890016345600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40637417 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2878760585997297 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2873580916555776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73742235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632675383748608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2893577867916800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325004807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2878760943586547 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632675365136384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376” 8¸3µ4ì9/ _wandbruntime‚16193  _runtime‚16193¥®LP'í9‹N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚16193.757265039 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19967801381373 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚18899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175103888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175103888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40639749 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652924464571392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325031484 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1557 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2873728174624768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632708793894912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843565350634165e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149927594 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175093897 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40639474 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19997963987968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2878908465312253 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2878908107722891 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149927594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632708812507136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175093897 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2873728157585408 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73746037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652924357389312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20215563494400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325031484 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20215652064256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ôiži9î9N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843565350634165e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149927594 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175093897 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40639474 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19997963987968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2878908465312253 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2878908107722891 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149927594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632708812507136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175093897 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2873728157585408 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73746037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652924357389312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20215563494400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325031484 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20215652064256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚16193.757265039 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19967801381373 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚18899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175103888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175103888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40639749 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652924464571392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325031484 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1557 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2873728174624768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632708793894912• ̪04ï9/ _wandbruntime‚16193  _runtime‚16193¸ ‡4ð9‰ 1loss/SRqqt1/detection_loss‚0.8849676069596221 1loss/SRqqt2/detection_loss‚0.5041939180007029 /loss/FBt1/detection_loss‚0.3269931714332071 /loss/FBt2/detection_loss‚0.3495686782537761 _runtime‚16193.759645102 3loss/FBt2/assignment_loss‚0.0006429762123445832 trainer/global_step‚18899 $ _timestamp‚1.784356535066338e+09 _step‚1558 0loss/FRt1/assignment_loss‚1.5405917275105423 /loss/FRt2/detection_loss‚1.1596571355347338 $loss/total_loss‚6.94960255690364 0loss/FRt2/assignment_loss‚0.7365765216578526 3loss/SRqqt2/assignment_loss‚0.44397473375750657 3loss/FBt1/assignment_loss‚1.710794428404554e-06 epoch‚47 3loss/SRqqt1/assignment_loss‚0.49978447888383215 /loss/FRt1/detection_loss‚0.5026498979050917w8RÒñ9Ž 0loss/FRt2/assignment_loss‚0.7365765216578526 3loss/SRqqt2/assignment_loss‚0.44397473375750657 3loss/FBt1/assignment_loss‚1.710794428404554e-06 epoch‚47 3loss/SRqqt1/assignment_loss‚0.49978447888383215 /loss/FRt1/detection_loss‚0.5026498979050917 1loss/SRqqt1/detection_loss‚0.8849676069596221 1loss/SRqqt2/detection_loss‚0.5041939180007029 /loss/FBt1/detection_loss‚0.3269931714332071 /loss/FBt2/detection_loss‚0.3495686782537761 _runtime‚16193.759645102 3loss/FBt2/assignment_loss‚0.0006429762123445832 trainer/global_step‚18899 $ _timestamp‚1.784356535066338e+09 _step‚1558 0loss/FRt1/assignment_loss‚1.5405917275105423 /loss/FRt2/detection_loss‚1.1596571355347338 $loss/total_loss‚6.94960255690364– üKKò9‚(:’ ½½ìÒܸœ“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Šï·3ó9‚(:¨ ½½ìÒ‡õÏ— network.recv‚ 4.303947e+06 disk.sda2.in‚ 82.544921875 network.sent‚ 3.0366051e+07$ disk.nvme0c0n1.in‚27633.02734375( disk.nvme0c0n1.out‚35216.67333984375 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27633.02734375 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚541.0478515625 disk.rbd0.in‚ 0.16015625( proc.memory.availableMB‚ 369.08203125# proc.memory.rssMB‚ 2806.44921875* proc.memory.percent‚17.129206657409668$ memory_percent‚97.74730205535889 cpu‚26.16642105467722) disk./.usagePercent‚39.30207825073042$ disk./.usageGB‚585.6293678283691 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1546.5625 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.dm-0.in‚ 27618.9921875& disk.nvme0n1.out‚35216.67333984375 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 50.4111328125 disk.sda3.out‚ 1546.5625 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚35216.67333984375 disk.sda.out‚ 1717.09765625 disk.sda3.in‚ 417.107421875 Ä^P‘ô9‚(:† ̽ìÒ†ò¼“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑãÚ4õ9‚(:© ̽ìÒŒËé•) disk./.usagePercent‚39.30209745129549 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚27633.02734375 disk.rbd1.in‚ 2.1796875 cpu‚26.166027151889253$ disk./.usageGB‚585.6296539306641 disk.dm-1.out‚ 1546.9453125$ disk.nvme0c0n1.in‚27633.02734375( proc.memory.availableMB‚ 369.14453125 disk.dm-1.in‚ 233.44140625& disk.nvme0c0n1.out‚35221.794921875 disk.sda.in‚541.0478515625 disk.rbd0.in‚ 0.16015625 disk.sda.out‚ 1717.48046875 disk.sda3.out‚ 1546.9453125 disk.sda1.in‚ 8.759765625 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 50.4111328125 network.recv‚ 4.307473e+06! disk.dm-0.out‚35221.794921875 disk.sda1.out‚0* proc.memory.percent‚17.129206657409668 disk.drbd2165.out‚0 disk.dm-0.in‚ 27618.9921875 disk.rbd0.out‚0$ memory_percent‚97.74692058563232# proc.memory.rssMB‚ 2806.44921875 network.sent‚ 3.0427875e+07 disk.sda2.out‚ 170.53515625$ disk.nvme0n1.out‚35221.794921875±û«Â‘ö9‚(:† Û½ìÒ÷ž©“ gpu.0.gpu‚93.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vN/ï7÷9‚(:¬ Û½ìÒ¢Áù• disk.sda1.in‚ 8.759765625 disk.sda3.out‚ 1547.41796875 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1547.41796875 disk.rbd0.out‚0 disk.sda.in‚541.0478515625& disk.nvme0n1.out‚35230.42041015625 disk.sda1.out‚0( disk.nvme0c0n1.out‚35230.42041015625 disk.sda.out‚ 1717.953125 disk.sda2.out‚ 170.53515625# proc.memory.rssMB‚ 2806.44921875) disk./.usagePercent‚39.30207927476056 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27755.69140625 disk.rbd1.in‚ 2.1796875' proc.memory.availableMB‚ 369.1484375 proc.cpu.threads‚17 network.sent‚ 3.0433462e+07# disk.dm-0.out‚35230.42041015625 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 50.4736328125" disk.nvme0n1.in‚27755.69140625 cpu‚26.165596777052656$ memory_percent‚97.74689674377441 network.recv‚ 4.311064e+06 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚ 27741.65625 disk.sda2.in‚ 82.544921875* proc.memory.percent‚17.129206657409668$ disk./.usageGB‚585.6293830871582cû'4ø9/ _wandbruntime‚16233  _runtime‚16233ëƒVÊù9Ä _runtime‚16233.203873455 _step‚1559 & lr-AdamW/pg1‚3.178531333605811e-06 & lr-AdamW/pg2‚3.178531333605811e-06 trainer/global_step‚18949 % _timestamp‚1.7843565745102978e+09n™©JÏú9É & lr-AdamW/pg1‚3.178531333605811e-06 & lr-AdamW/pg2‚3.178531333605811e-06 trainer/global_step‚18949 % _timestamp‚1.7843565745102978e+09 _runtime‚16233.203873455 _step‚1559— —é³¢4û9/ _wandbruntime‚16233  _runtime‚16233)ç (ü9€P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175549274 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40740824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2866119686852608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175539297 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20265860096512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33192071 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20265936090624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2900990614900224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2880942968037376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40741081 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73933152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚654612757765632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2880942968037376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175539297 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843565745113735e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2900990614900224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150301562 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20047646862848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚1560 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2880942985076736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2886137093667055 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16233.208342311 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2886136736077805 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325840859 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73932892 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2866119669813248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20047988201984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 trainer/global_step‚18949 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2900990973278720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚634346803062784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2900990614900224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2880942968037376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40741081 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73933152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚654612757765632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2880942968037376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175539297 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843565745113735e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2900990614900224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150301562 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20047646862848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚1560 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2880942985076736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2886137093667055 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16233.208342311 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2886136736077805 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325840859 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73932892 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2866119669813248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976˜ ÊØ”ö4þ9/ _wandbruntime‚16233  _runtime‚16233^¼A'ÿ9‹N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20049000836096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 trainer/global_step‚18949 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33193813 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150309194 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150309194 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1561 _runtime‚16233.819519205 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20048659489792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20266874737664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843565751257055e+09 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2886284615392761 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2901138868556800 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2881090226106368 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2866266197524480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33193816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2866266180485120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654647213741056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73936681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325867536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20018417868281 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325867536 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2881090209067008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325857543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20266963307520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2886284257803399 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚634380250433536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654647213741056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73936681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325867536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20018417868281 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325867536 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2881090209067008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325857543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20266963307520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2886284257803399 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚634380250433536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33193813 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150309194 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150309194 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1561 _runtime‚16233.819519205™ ž¢44:/ _wandbruntime‚16233  _runtime‚16233Ó¯üÀ‚:ˆ trainer/global_step‚18949 _runtime‚16233.821595233 2loss/SRqqt1/assignment_loss‚0.4449258690797744 3loss/FBt2/assignment_loss‚2.705421646423021e-05 1loss/SRqqt2/detection_loss‚0.5493858180247226 % _timestamp‚1.7843565751282737e+09 _step‚1562 0loss/FRt2/assignment_loss‚0.6416613185992006 /loss/FRt2/detection_loss‚1.0500820172957093 1loss/SRqqt1/detection_loss‚0.8392032340084561 3loss/SRqqt2/assignment_loss‚0.36523805035360396 3loss/FBt1/assignment_loss‚0.0002796135899863281 .loss/FRt1/detection_loss‚0.500662388753375 epoch‚47 /loss/FRt1/assignment_loss‚1.474738789943588 /loss/FBt1/detection_loss‚0.6708047471642763 /loss/FBt2/detection_loss‚0.3509538937018955 %loss/total_loss‚6.887962794731052ê†ö(ƒ: 3loss/SRqqt2/assignment_loss‚0.36523805035360396 3loss/FBt1/assignment_loss‚0.0002796135899863281 .loss/FRt1/detection_loss‚0.500662388753375 epoch‚47 /loss/FRt1/assignment_loss‚1.474738789943588 /loss/FBt1/detection_loss‚0.6708047471642763 /loss/FBt2/detection_loss‚0.3509538937018955 %loss/total_loss‚6.887962794731052 trainer/global_step‚18949 _runtime‚16233.821595233 2loss/SRqqt1/assignment_loss‚0.4449258690797744 3loss/FBt2/assignment_loss‚2.705421646423021e-05 1loss/SRqqt2/detection_loss‚0.5493858180247226 % _timestamp‚1.7843565751282737e+09 _step‚1562 0loss/FRt2/assignment_loss‚0.6416613185992006 /loss/FRt2/detection_loss‚1.0500820172957093 1loss/SRqqt1/detection_loss‚0.8392032340084561š ‡©áž„:‚(:“ ê½ìÒÁ÷—” gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Žáÿ1…:‚(:¦ ê½ìÒ¶½•–$ disk./.usageGB‚585.6294136047363 disk.drbd2165.out‚0 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27755.69140625 disk.rbd1.out‚ 50.4736328125 disk.sda1.in‚ 8.759765625 proc.cpu.threads‚17) disk./.usagePercent‚39.30208132282083 disk.sda.out‚ 1718.19140625 disk.sda2.in‚ 82.544921875$ memory_percent‚97.74677753448486( proc.memory.availableMB‚ 369.16796875# proc.memory.rssMB‚ 2806.44921875$ disk.nvme0c0n1.in‚27755.69140625( disk.nvme0c0n1.out‚35234.62451171875# disk.dm-0.out‚35234.62451171875 disk.rbd0.in‚ 0.16015625 network.sent‚ 3.043855e+07 disk.sda.in‚541.0478515625 disk.rbd1.in‚ 2.1796875 disk.sda3.out‚ 1547.65625 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚ 27741.65625& disk.nvme0n1.out‚35234.62451171875 network.recv‚ 4.314335e+06 disk.sda1.out‚0 disk.dm-1.out‚ 1547.65625 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625* proc.memory.percent‚17.129206657409668 cpu‚26.1651668866754S铆:‚(:ˆ ù½ìÒõƒÃ“ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Πñû6‡:‚(:« ù½ìÒ‡³ê• disk.drbd2165.in‚ 2.25390625$ disk.nvme0c0n1.in‚27755.69140625 disk.sda3.in‚ 417.107421875' disk.nvme0c0n1.out‚35239.9677734375 disk.drbd2165.out‚0 disk.sda.out‚ 1718.55859375% disk.nvme0n1.out‚35239.9677734375 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.dm-0.in‚ 27741.65625 disk.sda2.in‚ 82.544921875$ memory_percent‚97.74577617645264 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1548.0234375 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.sda3.out‚ 1548.0234375 disk.rbd1.in‚ 2.1796875* disk./.usagePercent‚39.302082346850966" disk.dm-0.out‚35239.9677734375 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27755.69140625 disk.rbd1.out‚ 50.5361328125# proc.memory.rssMB‚ 2806.44921875 cpu‚26.164755009397172( proc.memory.availableMB‚ 369.33203125 network.sent‚ 3.0501733e+07 network.recv‚ 4.318891e+06$ disk./.usageGB‚585.6294288635254 disk.rbd0.out‚0* proc.memory.percent‚17.129206657409668Ô„¢!4ˆ:/ _wandbruntime‚16273  _runtime‚16273BQò5̉:Æ trainer/global_step‚18999 % _timestamp‚1.7843566143470376e+09 _runtime‚16273.040608181 _step‚1563 ' lr-AdamW/pg1‚2.8839296407860398e-06 ' lr-AdamW/pg2‚2.8839296407860398e-06¡öJÑŠ:Ë trainer/global_step‚18999 % _timestamp‚1.7843566143470376e+09 _runtime‚16273.040608181 _step‚1563 ' lr-AdamW/pg1‚2.8839296407860398e-06 ' lr-AdamW/pg2‚2.8839296407860398e-06› –¤4‹:/ _wandbruntime‚16273  _runtime‚16273EÐi0Œ:€P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908403361883648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚18999 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656335181161984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20098683703808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326676911 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175993749 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150683162 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2888305036558336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2908403361883648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150683164 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1564 G>DeviceStatsMonitor.on_train_batch_start/a¿>1tÖctive.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2893512886158313 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20098342364672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74122713 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20067682751465 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176003726 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20098683703808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150683164 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2888305036558336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316940172800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636018240989184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 $ _timestamp‚1.784356614348108e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2888305019518976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908403361883648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚18999 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656335181161984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20098683703808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326676911 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175993749 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150683162 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2888305036558336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2908403361883648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150683164 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2893512886158313 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20098342364672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74122713 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20067682751465 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176003726 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20098683703808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150683164 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2888305036558336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316940172800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636018240989184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 $ _timestamp‚1.784356614348108e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2888305019518976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326703588 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843566149609995e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74126480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20099696337920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176002801 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20069034355189 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2888452260548608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2873591731118080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326703588 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2908551973925888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20317959000064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20099696337920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636051688359936 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20068693805187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74126212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636051669747712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318051764224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚16273.654739525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656369628747776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326693595 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2888452277587968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176012792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2873591714078720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176012792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636051688359936 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20068693805187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74126212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636051669747712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318051764224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚16273.654739525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656369628747776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326693595 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2888452277587968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176012792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2873591714078720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176012792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326703588 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843566149609995e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74126480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20099696337920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176002801 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20069034355189 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2888452260548608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2873591731118080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 aõ*4‘:/ _wandbruntime‚16273  _runtime‚16273_Â[ñ’:‰ %loss/total_loss‚6.752538635225132 _step‚1566 0loss/FRt2/assignment_loss‚0.6804278053916417 0loss/SRqqt2/detection_loss‚0.604759849893256 /loss/FBt2/detection_loss‚0.3216838212948172 epoch‚47 _runtime‚16273.658244391 4loss/FBt1/assignment_loss‚1.5140758495402196e-05 2loss/FBt2/assignment_loss‚0.000622529043817831 /loss/FRt2/detection_loss‚1.2460617519872008 % _timestamp‚1.7843566149647818e+09 0loss/FRt1/assignment_loss‚1.4856685125754654 /loss/FRt1/detection_loss‚0.5420003000302527 /loss/FBt1/detection_loss‚0.2997893203212243 trainer/global_step‚18999 3loss/SRqqt1/assignment_loss‚0.44682100263716007 2loss/SRqqt2/assignment_loss‚0.3552024559516959 1loss/SRqqt1/detection_loss‚0.7694861453401051=A©“:Ž 0loss/FRt1/assignment_loss‚1.4856685125754654 /loss/FRt1/detection_loss‚0.5420003000302527 /loss/FBt1/detection_loss‚0.2997893203212243 trainer/global_step‚18999 3loss/SRqqt1/assignment_loss‚0.44682100263716007 2loss/SRqqt2/assignment_loss‚0.3552024559516959 1loss/SRqqt1/detection_loss‚0.7694861453401051 %loss/total_loss‚6.752538635225132 _step‚1566 0loss/FRt2/assignment_loss‚0.6804278053916417 0loss/SRqqt2/detection_loss‚0.604759849893256 /loss/FBt2/detection_loss‚0.3216838212948172 epoch‚47 _runtime‚16273.658244391 4loss/FBt1/assignment_loss‚1.5140758495402196e-05 2loss/FBt2/assignment_loss‚0.000622529043817831 /loss/FRt2/detection_loss‚1.2460617519872008 % _timestamp‚1.7843566149647818e+09ž {µQ±ž”:‚(:“ ˆ¾ìÒºýº“ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.806' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.361200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02é÷2.•:‚(:£ ˆ¾ìÒÔ»þ• disk.sda1.in‚ 8.759765625 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚35242.869140625 network.sent‚ 3.0506818e+07 disk.dm-1.out‚ 1548.60546875 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1719.140625 network.recv‚ 4.322208e+06* disk./.usagePercent‚39.302086698979046 disk.rbd0.in‚ 0.16015625 disk.drbd2165.out‚0* proc.memory.percent‚17.129206657409668 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚27755.69140625 disk.rbd1.out‚ 50.5361328125& disk.nvme0c0n1.out‚35242.869140625 disk.dm-0.in‚ 27741.65625$ proc.memory.availableMB‚369.0625 proc.cpu.threads‚17 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0! disk.dm-0.out‚35242.869140625$ disk./.usageGB‚585.6294937133789 disk.sda3.out‚ 1548.60546875 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625$ disk.nvme0c0n1.in‚27755.69140625 disk.sda3.in‚ 417.107421875$ memory_percent‚97.74742126464844# proc.memory.rssMB‚ 2806.44921875 cpu‚26.16435755509282á5~–:‚(:’ —¾ìÒœ¼“ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÁUŠ&—:‚(:› —¾ìÒ´Œê• disk.nvme0n1.out‚ 35247.71875 disk.rbd0.out‚0$ memory_percent‚97.74744510650635 proc.cpu.threads‚17 network.recv‚ 4.327058e+06 disk.dm-0.out‚ 35247.71875 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625( proc.memory.availableMB‚ 369.05859375$ disk./.usageGB‚585.6300964355469 disk.sda3.out‚ 1549.015625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27905.49609375 disk.rbd1.in‚ 2.1796875# proc.memory.rssMB‚ 2806.44921875) disk./.usagePercent‚39.30212714816946 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.dm-1.out‚ 1549.015625* proc.memory.percent‚17.129206657409668 cpu‚26.163931258390658 network.sent‚ 3.0570006e+07" disk.nvme0n1.in‚27905.49609375 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 50.5986328125" disk.nvme0c0n1.out‚ 35247.71875 disk.dm-0.in‚ 27891.4609375 disk.sda.in‚541.0478515625 disk.sda.out‚ 1719.55078125 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.drbd2165.in‚ 2.25390625©Û¿˜:‚(:’ ¦¾ìÒ¬‹¶“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.287999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¾‘82™:‚(:§ ¦¾ìÒ Ñ• disk.dm-0.in‚ 27891.4609375" disk.nvme0n1.in‚27905.49609375 disk.rbd1.in‚ 2.1796875 network.recv‚ 4.33044e+06) disk./.usagePercent‚39.30214378865919 disk.rbd1.out‚ 50.5986328125' proc.memory.availableMB‚ 369.2421875 network.sent‚ 3.0575102e+07 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1549.4453125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27905.49609375& disk.nvme0c0n1.out‚35250.755859375 disk.sda2.in‚ 82.544921875$ disk.nvme0n1.out‚35250.755859375 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1549.4453125$ memory_percent‚97.74632453918457# proc.memory.rssMB‚ 2806.44921875 proc.cpu.threads‚17$ disk./.usageGB‚585.6303443908691 disk.rbd0.in‚ 0.16015625 cpu‚26.163505867244442 disk.drbd2165.in‚ 2.25390625! disk.dm-0.out‚35250.755859375 disk.sda.out‚ 1719.98046875* proc.memory.percent‚17.129206657409668 disk.sda1.out‚0 disk.sda.in‚541.0478515625­@Å4š:/ _wandbruntime‚16313  _runtime‚16313OP»Ë›:Å ' lr-AdamW/pg1‚2.6035644049835785e-06 ' lr-AdamW/pg2‚2.6035644049835785e-06 trainer/global_step‚19049 % _timestamp‚1.7843566544083965e+09 _runtime‚16313.10200192 _step‚1567ú~ÐZÐœ:Ê _step‚1567 ' lr-AdamW/pg1‚2.6035644049835785e-06 ' lr-AdamW/pg2‚2.6035644049835785e-06 trainer/global_step‚19049 % _timestamp‚1.7843566544083965e+09 _runtime‚16313.10200192Ÿ —ÉkÞ4:/ _wandbruntime‚16313  _runtime‚16313“ûŠã(ž:P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74312402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33366568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20368078456320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2900889036238821 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118299238373 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915816467245568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16313.103380171 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637689697527808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2895667088039936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2895667071000576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20367989879296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843566544095337e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40945549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚658057775984128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658057668794880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40945831 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151064762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151064762 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637689678915584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118639788263 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20149379205632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2880770754039808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151064764 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20149379205632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19049 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2915816467245568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176448201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176448201 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2915816108867072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176458178 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20149037866496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20149037866496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2895667071000576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915816108867072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2895667088039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327522942 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151064764 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0”Œo (Ÿ:†P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2915816467245568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176448201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176448201 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2915816108867072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176458178 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20149037866496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20149037866496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2895667071000576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915816108867072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2895667088039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327522942 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151064764 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74312402 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33366568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20368078456320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2900889036238821 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118299238373 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915816467245568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16313.103380171 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637689697527808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2895667088039936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2895667071000576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20367989879296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843566544095337e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40945549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚658057775984128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658057668794880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40945831 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151064762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151064762 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637689678915584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118639788263 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20149379205632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2880770754039808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151064764 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20149379205632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19049  yS4 :/ _wandbruntime‚16313  _runtime‚16313Ñùµ•$ ¡:‹N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2901036915553777 % _timestamp‚1.7843566550236669e+09 _runtime‚16313.717440213 _step‚1569 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176457253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20119650842097 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20119310292095 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2880917247672320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20369005569024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20150391839744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2915964720909312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74316152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33368313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40947836 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2901036557964415 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176467244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2880917264711680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176467244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327529647 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658092226716672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40947580 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2895814329069568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74315893 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327539640roÀŽ'¢:N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2915964720909312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74316152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33368313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40947836 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2901036557964415 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176467244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2880917264711680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176467244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327529647 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658092226716672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40947580 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2895814329069568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74315893 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327539640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2901036915553777 % _timestamp‚1.7843566550236669e+09 _runtime‚16313.717440213 _step‚1569 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176457253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20119650842097 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20119310292095 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2880917247672320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20369005569024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20150391839744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088¡ îk Ö4£:/ _wandbruntime‚16313  _runtime‚16313XÁ>¤:‹ % _timestamp‚1.7843566550267644e+09 /loss/FRt1/detection_loss‚0.4919723776538491 /loss/FRt2/detection_loss‚1.1327142210383339 0loss/SRqqt1/detection_loss‚0.919804313021863 &loss/total_loss‚7.1204879726368535 trainer/global_step‚19049 3loss/SRqqt1/assignment_loss‚0.39229981444651185 3loss/FBt1/assignment_loss‚5.976683280942811e-06 1loss/SRqqt2/detection_loss‚0.5236236626521492 /loss/FBt1/detection_loss‚0.5796483645942223 /loss/FBt2/detection_loss‚0.3266812581637023 0loss/FRt1/assignment_loss‚1.5631246738684805 0loss/FRt2/assignment_loss‚0.7791607114760315 3loss/SRqqt2/assignment_loss‚0.40958087602423354 3loss/FBt2/assignment_loss‚0.0018717230141953678 _runtime‚16313.720050059 _step‚1570 epoch‚47 ]èD¥: /loss/FRt1/detection_loss‚0.4919723776538491 /loss/FRt2/detection_loss‚1.1327142210383339 0loss/SRqqt1/detection_loss‚0.919804313021863 &loss/total_loss‚7.1204879726368535 trainer/global_step‚19049 3loss/SRqqt1/assignment_loss‚0.39229981444651185 3loss/FBt1/assignment_loss‚5.976683280942811e-06 1loss/SRqqt2/detection_loss‚0.5236236626521492 /loss/FBt1/detection_loss‚0.5796483645942223 /loss/FBt2/detection_loss‚0.3266812581637023 0loss/FRt1/assignment_loss‚1.5631246738684805 0loss/FRt2/assignment_loss‚0.7791607114760315 3loss/SRqqt2/assignment_loss‚0.40958087602423354 3loss/FBt2/assignment_loss‚0.0018717230141953678 _runtime‚16313.720050059 _step‚1570 epoch‚47 % _timestamp‚1.7843566550267644e+09¢ HY™‘¦:‚(:† µ¾ìÒùºÖ“ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ʋ­:§:‚(:¯ µ¾ìÒïÓÛ– network.recv‚ 4.333843e+06 disk.sda1.out‚0 disk.sda.in‚541.0478515625$ disk.nvme0c0n1.in‚27905.49609375 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875$ disk./.usageGB‚585.6303596496582 disk.rbd0.in‚ 0.16015625( proc.memory.availableMB‚ 368.85546875 cpu‚26.16309231772991 network.sent‚ 3.0580146e+07 disk.dm-1.out‚ 1549.80078125 disk.rbd1.in‚ 2.1796875* proc.memory.percent‚17.129206657409668 disk.dm-1.in‚ 233.44140625 disk.sda.out‚ 1720.3359375 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 50.6611328125 disk.drbd2165.in‚ 2.25390625( disk.nvme0c0n1.out‚35255.19970703125& disk.nvme0n1.out‚35255.19970703125 disk.rbd0.out‚0$ memory_percent‚97.74868488311768 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚35255.19970703125" disk.nvme0n1.in‚27905.49609375 disk.sda3.out‚ 1549.80078125# proc.memory.rssMB‚ 2806.44921875) disk./.usagePercent‚39.30214481268932 disk.drbd2165.out‚0 disk.dm-0.in‚ 27891.4609375 proc.cpu.threads‚17|"D‘¨:‚(:† ľìÒø¸ì” gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=爇0©:‚(:¥ ľìÒ˜æú– disk.sda.in‚541.0478515625 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0( proc.memory.availableMB‚ 368.86328125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚27905.49609375 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚35257.80859375 disk.rbd1.out‚ 50.6611328125 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1550.15234375$ memory_percent‚97.74863719940186* proc.memory.percent‚17.129206657409668 cpu‚26.162697539645464 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1550.15234375 network.recv‚ 4.338167e+06 disk.dm-0.in‚ 27891.4609375 disk.dm-0.out‚35257.80859375 disk.sda2.out‚ 170.53515625# disk.nvme0n1.out‚35257.80859375 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0* disk./.usagePercent‚39.302162221201655# proc.memory.rssMB‚ 2806.44921875 network.sent‚ 3.0642733e+07 disk.sda.out‚ 1720.6875" disk.nvme0n1.in‚27905.49609375 disk.drbd2165.in‚ 2.25390625 disk.dm-1.in‚ 233.44140625$ disk./.usageGB‚585.6306190490723W®P–‘ª:‚(:† Ó¾ìÒ±œì“ gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨n¬5«:‚(:ª Ó¾ìÒ¹½ë–* disk./.usagePercent‚39.302152236907816 disk.dm-1.out‚ 1550.546875# disk.dm-0.out‚35262.60009765625 disk.sda3.out‚ 1550.546875 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0$ memory_percent‚97.74870872497559 disk.dm-1.in‚ 233.44140625 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚35262.60009765625 disk.rbd0.in‚ 0.16015625 cpu‚26.162289460112603 proc.cpu.threads‚17$ disk./.usageGB‚585.6304702758789 disk.sda.in‚541.0478515625 disk.sda.out‚ 1721.08203125 disk.rbd1.out‚ 50.7861328125 disk.sda1.in‚ 8.759765625' proc.memory.availableMB‚ 368.8515625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 28070.265625( disk.nvme0c0n1.out‚35262.60009765625 disk.dm-0.in‚28056.23046875 disk.nvme0n1.in‚ 28070.265625# proc.memory.rssMB‚ 2806.44921875 network.recv‚ 4.341753e+06 disk.drbd2165.in‚ 2.25390625 disk.sda3.in‚ 417.107421875 disk.sda1.out‚0 network.sent‚ 3.0647783e+07 disk.sda2.in‚ 82.544921875* proc.memory.percent‚17.129206657409668ÝYú4¬:/ _wandbruntime‚16352  _runtime‚16352³|QÀÌ­:Æ ' lr-AdamW/pg2‚2.3374536335336535e-06 trainer/global_step‚19099 % _timestamp‚1.7843566942777283e+09 _runtime‚16352.971330058 _step‚1571 ' lr-AdamW/pg1‚2.3374536335336535e-06ç‘5Ñ®:Ë ' lr-AdamW/pg1‚2.3374536335336535e-06 ' lr-AdamW/pg2‚2.3374536335336535e-06 trainer/global_step‚19099 % _timestamp‚1.7843566942777283e+09 _runtime‚16352.971330058 _step‚1571£ ñÏk4¯:/ _wandbruntime‚16352  _runtime‚16352.å ÅÛ!°:P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2923228855850496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2923229214228992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151446364 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2908265543908579 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1572 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2903029122482176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328358994 trainer/global_step‚19099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20199733368320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176912630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33453821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33453818 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151446362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74502389 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843566942788675e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176902653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903029122482176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151446364 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659780361537536 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2903029139521536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20419141700608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2888096270594048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2908265186319329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923229214228992 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923228855850496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.smal䯋Ð,l_pool.allocated‚20200074707456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20419226083328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74502662 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639361135454208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2888096287633408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20169256275171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659780258542592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328349015ëjXÌ (±:†P E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2908265186319329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923229214228992 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923228855850496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20200074707456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20419226083328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74502662 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639361135454208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2888096287633408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20169256275171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659780258542592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328349015 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2923228855850496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2923229214228992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151446364 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2908265543908579 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1572 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2903029122482176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328358994 trainer/global_step‚19099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20199733368320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176912630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33453821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33453818 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151446362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74502389 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843566942788675e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176902653 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903029122482176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151446364 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659780361537536 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2903029139521536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20419141700608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2888096270594048¤ ‚Ï'14²:/ _wandbruntime‚16353  _runtime‚163534¢äK'³:‹N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2908412708044923 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2923377109507072 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903176363511808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176911705 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _step‚1573 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2923377467892736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33455566 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2888242798305280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2903176363511808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843566948936212e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328365699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41050895 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151453994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20201087341568 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176921696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41050631 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20420154617344 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74506194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20169926779003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151453994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20201087341568 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20420247381504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151453996 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33455563 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328375692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328365699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151453996 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚19099 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176921696 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2923377467892736 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2923377109507072 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903176363511808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176911705 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _step‚1573 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2923377467892736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33455566 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2888242798305280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2903176363511808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843566948936212e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328365699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41050895 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151453994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20201087341568 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176921696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41050631 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20420154617344 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74506194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20169926779003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151453994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20201087341568 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20420247381504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151453996 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33455563 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328375692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328365699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151453996 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚19099 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176921696 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2923377467892736 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177367082 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41151410 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74692753 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2910391173963776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚661502878233600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329185067 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151827962 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2910391173963776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641032573380608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20219532212189 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20250428870144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151827964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177357105 _runtime‚16392.989361612 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641032554768384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20250428870144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20250770209280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930641602833920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329195046 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177357105 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74692478 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2895421821227008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2915641336399837 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2895421804187648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2930641961212416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚661502977034240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2930641961212416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169² Èç (Á:…P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843567342925174e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2930641602833920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20250770209280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177367082 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41151410 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74692753 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2910391173963776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚661502878233600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329185067 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151827962 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2910391173963776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641032573380608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20219532212189 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20250428870144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151827964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177357105 _runtime‚16392.989361612 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641032554768384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20250428870144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20250770209280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930641602833920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329195046 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177357105 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74692478 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2895421821227008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2915641336399837 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2895421804187648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2930641961212416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚661502977034240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2930641961212416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930789856490496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74696578 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41153762 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151835594 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41153498 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20471345855488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74696311 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329211744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151835594 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329201751 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2895568331898880 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29105384320327¾$:k68 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329211744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151835596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2910538432032768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20251441497088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20471432328192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚19149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20251782843392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33542816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2915789215714793 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1577 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74696311 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329211744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151835594 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329201751 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2895568331898880 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2910538432032768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329211744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151835596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2910538432032768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20251441497088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20471432328192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚19149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20251782843392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33542816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2915789215714793 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1577 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930789856490496© Uà— 4Å:/ _wandbruntime‚16393  _runtime‚16393ïÄŸÆ:‰ 0loss/FRt1/assignment_loss‚1.4430088497325382 2loss/SRqqt1/assignment_loss‚0.3715317868032032 /loss/FRt2/detection_loss‚1.1453657649406042 1loss/SRqqt2/detection_loss‚0.5340139502017207 -loss/FBt2/detection_loss‚0.41291146779075 %loss/total_loss‚6.829633653215785 epoch‚47 _runtime‚16393.602964375 2loss/SRqqt2/assignment_loss‚0.4028682642724553 3loss/FBt1/assignment_loss‚3.818615886516376e-06 /loss/FRt1/detection_loss‚0.4986814518404153 0loss/FBt1/detection_loss‚0.41631502584879915 % _timestamp‚1.7843567349096653e+09 _step‚1578 0loss/FRt2/assignment_loss‚0.7543978817500029 4loss/FBt2/assignment_loss‚6.0205342331244085e-05 1loss/SRqqt1/detection_loss‚0.8504751860770777 trainer/global_step‚191491õ!ÍÇ:Ž /loss/FRt1/detection_loss‚0.4986814518404153 0loss/FBt1/detection_loss‚0.41631502584879915 % _timestamp‚1.7843567349096653e+09 _step‚1578 0loss/FRt2/assignment_loss‚0.7543978817500029 4loss/FBt2/assignment_loss‚6.0205342331244085e-05 1loss/SRqqt1/detection_loss‚0.8504751860770777 trainer/global_step‚19149 0loss/FRt1/assignment_loss‚1.4430088497325382 2loss/SRqqt1/assignment_loss‚0.3715317868032032 /loss/FRt2/detection_loss‚1.1453657649406042 1loss/SRqqt2/detection_loss‚0.5340139502017207 -loss/FBt2/detection_loss‚0.41291146779075 %loss/total_loss‚6.829633653215785 epoch‚47 _runtime‚16393.602964375 2loss/SRqqt2/assignment_loss‚0.4028682642724553 3loss/FBt1/assignment_loss‚3.818615886516376e-06ª lkžÈ:‚(:“ €¿ìÒ˜›¦“ gpu.0.gpu‚51.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚255.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.068599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¨a3É:‚(:¨ €¿ìÒ×͉–$ disk./.usageGB‚585.6305046081543 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625# proc.memory.rssMB‚ 2806.44921875 disk.dm-0.in‚28056.23046875 disk.dm-1.in‚ 233.44140625( proc.memory.availableMB‚ 368.82421875) disk./.usagePercent‚39.30215454097563" disk.nvme0c0n1.in‚ 28070.265625' disk.nvme0c0n1.out‚35273.8310546875 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1551.66796875 proc.cpu.threads‚17 disk.rbd1.in‚ 2.1796875 cpu‚26.16104054957077% disk.nvme0n1.out‚35273.8310546875 disk.rbd1.out‚ 50.8486328125* proc.memory.percent‚17.129206657409668 network.sent‚ 3.0721137e+07 disk.sda.out‚ 1722.203125 disk.sda2.out‚ 170.53515625 disk.dm-1.out‚ 1551.66796875 disk.drbd2165.out‚0$ memory_percent‚97.74887561798096" disk.dm-0.out‚35273.8310546875 disk.sda.in‚541.0478515625 disk.nvme0n1.in‚ 28070.265625 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 network.recv‚ 4.352805e+06=òQ@‘Ê:‚(:† ¿ìÒÒÑ¥“ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Ê]!.Ë:‚(:£ ¿ìÒ¢¹¸• disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚35278.970703125! disk.dm-0.out‚35278.970703125$ memory_percent‚97.74837493896484 cpu‚26.160652412911674 proc.cpu.threads‚17 disk.sda.out‚ 1722.48828125 disk.dm-1.in‚ 233.44140625* proc.memory.percent‚17.129206657409668 network.sent‚ 3.0784757e+07 disk.sda2.out‚ 170.53515625 disk.sda1.in‚ 8.759765625 disk.drbd2165.in‚ 2.25390625! disk.nvme0n1.in‚ 28192.1640625$ disk.nvme0n1.out‚35278.970703125 disk.sda3.in‚ 417.107421875 network.recv‚ 4.357403e+06 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1551.953125 disk.rbd1.out‚ 50.9111328125 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1551.953125% proc.memory.availableMB‚ 368.90625) disk./.usagePercent‚39.30165993441947 disk.sda2.in‚ 82.544921875# disk.nvme0c0n1.in‚ 28192.1640625# proc.memory.rssMB‚ 2806.44921875$ disk./.usageGB‚585.6231346130371 disk.dm-0.in‚28178.12890625 disk.rbd1.in‚ 2.1796875nçhŠžÌ:‚(:“ ž¿ìÒ܆ԓ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.191200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aáúë6Í:‚(:« ž¿ìÒÀó„–# proc.memory.rssMB‚ 2806.44921875 network.sent‚ 3.0789841e+07& disk.nvme0n1.out‚35283.68212890625 disk.sda3.out‚ 1552.2734375* proc.memory.percent‚17.129206657409668 disk.sda.out‚ 1722.80859375 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625 disk.rbd0.in‚ 0.16015625% proc.memory.availableMB‚ 368.46875 cpu‚26.160245660307634# disk.nvme0c0n1.in‚ 28192.1640625! disk.nvme0n1.in‚ 28192.1640625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 network.recv‚ 4.360575e+06 disk.dm-1.in‚ 233.44140625 disk.sda2.in‚ 82.544921875 disk.sda1.in‚ 8.759765625# disk./.usageGB‚585.623119354248 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚35283.68212890625 disk.dm-0.in‚28178.12890625 disk.sda2.out‚ 170.53515625 disk.rbd1.out‚ 50.9111328125# disk.dm-0.out‚35283.68212890625 disk.rbd0.out‚0$ memory_percent‚97.75104522705078* disk./.usagePercent‚39.301658910389335 disk.dm-1.out‚ 1552.2734375 disk.sda.in‚541.0478515625]øív4Î:/ _wandbruntime‚16433  _runtime‚16433|{ÌÏ:Æ _runtime‚16433.094281785 _step‚1579 ' lr-AdamW/pg1‚1.8480629342463396e-06 ' lr-AdamW/pg2‚1.8480629342463396e-06 trainer/global_step‚19199 % _timestamp‚1.7843567744006255e+09fGÂèÑÐ:Ë _step‚1579 ' lr-AdamW/pg1‚1.8480629342463396e-06 ' lr-AdamW/pg2‚1.8480629342463396e-06 trainer/global_step‚19199 % _timestamp‚1.7843567744006255e+09 _runtime‚16433.094281785« AªpŸ4Ñ:/ _wandbruntime‚16433  _runtime‚16433ð}KP(Ò:€P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177821534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177811557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41253903 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2923017844069595 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚19199 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20270148699097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938054349817344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1580 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74881970 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2938054708195840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663225573001728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20270489248987 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2902747337781248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20301465711104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917753225445376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20301124371968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚16433.095526416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177811557 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917753242484736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20521481506304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642703992694784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2923017486480345 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20301124371968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152209564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20301465711104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74882224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33628321 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2938054708195840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152209562 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177821534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917753225445376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2938054349817344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152209562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663225474201088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330021119 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20301465711104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74882224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33628321 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2938054708195840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152209562 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177821534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917753225445376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2938054349817344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152209562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663225474201088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330021119 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938054349817344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1580 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74881970 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2938054708195840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663225573001728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20270489248987 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2902747337781248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20301465711104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917753225445376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20301124371968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚16433.095526416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177811557 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917753242484736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20521481506304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642703992694784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2923017486480345 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20301124371968¬ ›WË4Ô:/ _wandbruntime‚16433  _runtime‚16433gÒoÝ'Õ:ŠN PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2923165365795301 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330037803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41255927 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2938202961859584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642737421453312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20302478345216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2938202961859584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152217194 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74885735 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2902893865492480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917900466475008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784356775017852e+09 _step‚1581 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152217196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177820609 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330047796 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177830600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663260028450816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330047796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177820609 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152217194 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642737440065536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20522588385280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚19199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20302136998912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ãß‹'Ö:N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330037803 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330047796 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177830600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663260028450816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330047796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177820609 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152217194 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642737440065536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20522588385280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚19199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20302136998912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2923165365795301 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330037803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41255927 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2938202961859584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642737421453312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20302478345216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2938202961859584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152217194 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74885735 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2902893865492480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917900466475008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784356775017852e+09 _step‚1581 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152217196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177820609 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13577887415885925 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2511257529258728 +jet/accuracy_4_of_4‚0.01670061983168125 /particle/accuracy_3_of_4‚0.9968254566192627 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373708844184875 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SR`Ëî¡@2qqt0FBt/SRqqt_purity‚0.6679878830909729 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6384100914001465 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6546463966369629 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_1‚0.3464047908782959 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7148434519767761 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2827920615673065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08906424790620804 /particle/accuracy_4_of_4‚0.9479283690452576 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2937345504760742 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7680426836013794 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22357234358787537 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7750258445739746 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.37938761711120605 _step‚1583 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2129552662372589 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04620777815580368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931967973709106 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 +particle/specificity‚0.8984387516975403 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2826176583766937 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.785014808177948 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7877386808395386 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23947246372699738 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6331213712692261 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_3_of_3‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7094250917434692 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6049227714538574 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6833277940750122 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_3_of_3‚0.03792091831564903 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15238791704177856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23400749266147614 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6571041941642761 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7235599756240845 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 epoch‚47 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5494677424430847 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1449648141860962 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3379855155944824 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.329414039850235 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2123039960861206 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10002398490905762 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6538277864456177 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2449372559785843 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23625348508358002 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14998415112495422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7278473973274231 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5695310831069946 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6918545365333557 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3632279634475708 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2491438090801239 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373708844184875 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09780268371105194 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29335805773735046 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03633022680878639 *jet/accuracy_1_of_2‚0.7858832478523254 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27966251969337463 particle/accuracy_4_of_5‚1 (jet/accuracy_1_of_3‚0.92719566822052 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6117768287658691 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37653011083602905 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2403750866651535 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05178915709257126 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6279239654541016 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 _runtime‚16450.828683232 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2480369359254837 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16681186854839325 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9806751012802124 )loss/val_total_loss‚7.700031874844386 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.5123410224914551 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2828385829925537 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6907112002372742 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7275498509407043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7329548597335815 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6848543882369995 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.000186011908226646488Ï´†‚=Þ:üz *jet/accuracy_1_of_4‚0.9806751012802124 )loss/val_total_loss‚7.700031874844386 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_3‚0.5123410224914551 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2828385829925537 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6907112002372742 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7275498509407043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7329548597335815 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6848543882369995 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_2‚0.1259116679430008 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03611055016517639 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6419703364372253 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2972862422466278 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8332016468048096 jet/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.567559540271759 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2827920615673065 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 +particle/sensitivity‚0.5946555137634277 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5336311459541321 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5777705311775208 +jet/accuracy_3_of_4‚0.04078979045152664 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 trainer/global_step‚19199 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6533946394920349 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23782634735107422 % _timestamp‚1.7843567921343083e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8287604451179504 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2744535803794861 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7076824307441711 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.669255793094635 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31507429480552673 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782057166099548 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13577887415885925 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2511257529258728 +jet/accuracy_4_of_4‚0.01670061983168125 /particle/accuracy_3_of_4‚0.9968254566192627 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373708844184875 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6679878830909729 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6384100914001465 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6546463966369629 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_1‚0.3464047908782959 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7148434519767761 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2827920615673065 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08906424790620804 /particle/accuracy_4_of_4‚0.9479283690452576 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2937345504760742 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7680426836013794 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22357234358787537 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7750258445739746 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.37938761711120605 _step‚1583 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2129552662372589 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04620777815580368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931967973709106 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 +particle/specificity‚0.8984387516975403 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2826176583766937 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.785014808177948 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7877386808395386 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23947246372699738 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6331213712692261 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_3_of_3‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7094250917434692 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6049227714538574 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6833277940750122 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_3_of_3‚0.03792091831564903 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15238791704177856 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23400749266147614 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6571041941642761 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7235599756240845 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 epoch‚47 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5494677424430847 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1449648141860962 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3379855155944824 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.329414039850235 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2123039960861206 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10002398490905762 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6538277864456177 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2449372559785843 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23625348508358002 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14998415112495422 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7278473973274231 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5695310831069946 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6918545365333557 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3632279634475708 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2491438090801239 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373708844184875 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09780268371105194 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29335805773735046 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03633022680878639 *jet/accuracy_1_of_2‚0.7858832478523254 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27966251969337463 particle/accuracy_4_of_5‚1 (jet/accuracy_1_of_3‚0.92719566822052 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6117768287658691 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37653011083602905 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2403750866651535 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05178915709257126 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6279239654541016 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 _runtime‚16450.828683232 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2480369359254837 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16681186854839325 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 ("Purity/2FRt2SRqqt*FBt/event_purity‚0¯ úNAêžß:‚(:“ ¼¿ìÒÔÁÓ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚249.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.830200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00?8à:‚(:­ ¼¿ìÒÒØÿ• disk.sda.in‚541.0478515625 disk.dm-1.out‚ 1553.10546875# memory_percent‚97.7672815322876 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0 disk.dm-1.in‚ 233.44140625* proc.memory.percent‚17.145228385925293# disk.nvme0c0n1.in‚ 28192.1640625 disk.sda.out‚ 1723.640625 disk.rbd0.in‚ 0.16015625 proc.cpu.threads‚17# disk.dm-0.out‚35291.92431640625 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.sda1.out‚0 network.sent‚ 3.0857734e+07 network.recv‚ 4.367995e+06$ disk./.usageGB‚585.6306457519531 disk.sda2.out‚ 170.53515625) disk./.usagePercent‚39.30216401325439( disk.nvme0c0n1.out‚35291.92431640625 disk.dm-0.in‚28178.12890625! disk.nvme0n1.in‚ 28192.1640625& disk.nvme0n1.out‚35291.92431640625 cpu‚26.163548592887487 disk.sda3.out‚ 1553.10546875 disk.rbd1.out‚ 50.9736328125 disk.sda1.in‚ 8.759765625# proc.memory.rssMB‚ 2809.07421875( proc.memory.availableMB‚ 365.80859375þçyžá:‚(:“ Ë¿ìÒ„×Ý• gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÚ5.â:‚(:£ Ë¿ìÒȘߗ network.sent‚ 3.0915006e+07 disk.dm-1.in‚ 233.44140625! disk.nvme0n1.in‚ 28326.9921875 cpu‚26.16313651880607$ disk./.usageGB‚585.6306953430176 disk.sda.out‚ 1724.1953125* proc.memory.percent‚17.145228385925293) disk./.usagePercent‚39.30216734135234 disk.dm-0.in‚28312.95703125 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1553.66015625 disk.rbd1.out‚ 51.0361328125 disk.rbd0.in‚ 0.16015625# proc.memory.rssMB‚ 2809.07421875 network.recv‚ 4.371939e+06% disk.nvme0c0n1.out‚35295.74609375 disk.dm-0.out‚35295.74609375# disk.nvme0n1.out‚35295.74609375 proc.cpu.threads‚17 disk.sda.in‚541.0478515625 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.sda1.in‚ 8.759765625$ memory_percent‚97.76811599731445 disk.dm-1.out‚ 1553.66015625# disk.nvme0c0n1.in‚ 28326.9921875 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 disk.sda2.out‚ 170.53515625 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0& proc.memory.availableMB‚ 365.671875$þ›Ÿã:‚(:’ Ú¿ìÒ ¬§“ gpu.0.gpu‚30.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.636599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fM†Çä:‚(:­ Ú¿ìÒ®¹Ç•$ disk./.usageGB‚585.6306228637695 disk.dm-1.in‚ 233.44140625& disk.nvme0n1.out‚35299.22021484375 disk.sda3.in‚ 417.107421875 disk.rbd1.out‚ 51.0361328125( proc.memory.availableMB‚ 365.51171875 network.recv‚ 4.375113e+06 disk.rbd1.in‚ 2.1796875 cpu‚26.16273826083452 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1553.8828125 network.sent‚ 3.0920071e+07) disk./.usagePercent‚39.30216247720919 disk.rbd0.out‚0 disk.dm-1.out‚ 1553.8828125 disk.dm-0.in‚28312.95703125# disk.dm-0.out‚35299.22021484375 disk.drbd2165.in‚ 2.25390625# proc.memory.rssMB‚ 2809.07421875* proc.memory.percent‚17.145228385925293 proc¾·“•q.cpu.threads‚17 disk.sda.out‚ 1724.41796875 disk.sda2.out‚ 170.53515625 disk.sda2.in‚ 82.544921875$ memory_percent‚97.76909351348877! disk.nvme0n1.in‚ 28326.9921875 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.sda1.in‚ 8.759765625# disk.nvme0c0n1.in‚ 28326.9921875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚35299.22021484375Ѿž°4å:/ _wandbruntime‚16490  _runtime‚164903QAÉæ:à # _timestamp‚1.78435683160252e+09 _runtime‚16490.29613296 _step‚1584 ' lr-AdamW/pg1‚1.6248144390590611e-06 ' lr-AdamW/pg2‚1.6248144390590611e-06 trainer/global_step‚19249GñôÎç:È ' lr-AdamW/pg1‚1.6248144390590611e-06 ' lr-AdamW/pg2‚1.6248144390590611e-06 trainer/global_step‚19249 # _timestamp‚1.78435683160252e+09 _runtime‚16490.29613296 _step‚1584° Q û4è:/ _wandbruntime‚16490  _runtime‚16490›?óF(é:ÿO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20369553079296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664982035390464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2947414755035136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41389348 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2927044860616704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644391361333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20590692669440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20338484911234 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33718088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178336331 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20369553079296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2927044843577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178346308 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚16490.29750518 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2947414755035136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152711830 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2932320377089412 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2911981551628288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947414396656640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331058138 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2932320019500162 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152711828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20590785440768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331048159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33718091 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784356831603779e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2911981534588928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1585 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2947414396656640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2927044843577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152711830 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664982146774016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75107436 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20369894418432 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75107694 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169Ó8²} (ê:„P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152711828 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178346308 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20369894418432 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644391342721024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41389603 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947414396656640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331058138 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2932320019500162 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152711828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20590785440768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331048159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33718091 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784356831603779e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2911981534588928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1585 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2947414396656640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2927044843577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152711830 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664982146774016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75107436 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20369894418432 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75107694 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20369553079296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664982035390464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2947414755035136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41389348 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2927044860616704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644391361333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20590692669440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20338484911234 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33718088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178336331 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20369553079296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2927044843577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178346308 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚16490.29750518 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2947414755035136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152711830 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2932320377089412 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2911981551628288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20339836514958 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178345383 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152719460 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20370907052544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2947563008698880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178345383 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932467898815118 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644424790091776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚16490.914540778 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20591720380928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178355374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75111230 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331074836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2927192084606976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75111501 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1586 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152719460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41391665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20370565706240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2912128062300160 % _timestamp‚1.7843568322206993e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331064843 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331064843 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331074836 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20339495964956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚19249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20370565706240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33719836 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665016491860480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152719462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2912128045260800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2927192084606976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438%p·×öí:N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331074836 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20339495964956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚19249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20370565706240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33719836 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665016491860480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152719462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2912128045260800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2927192084606976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20339836514958 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178345383 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152719460 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20370907052544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2947563008698880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178345383 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932467898815118 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644424790091776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚16490.914540778 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20591720380928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178355374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75111230 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331074836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2927192084606976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75111501 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1586 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152719460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41391665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20370565706240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2912128062300160 % _timestamp‚1.7843568322206993e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331064843 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331064843 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088² ø/îÒ4î:/ _wandbruntime‚16490  _runtime‚16490‡TÒþ ï:‡ 1loss/SRqqt1/detection_loss‚0.8718075464396414 /loss/FBt1/detection_loss‚0.2652494969687333 _step‚1587 3loss/SRqqt1/assignment_loss‚0.37291254826573816 3loss/FBt2/assignment_loss‚3.677217315788817e-06 /loss/FRt1/detection_loss‚0.5379895444188305 0loss/FBt2/detection_loss‚0.34197137067061945 %loss/total_loss‚6.823807794532574 -loss/FRt1/assignment_loss‚1.4812745237845 0loss/FRt2/assignment_loss‚0.7324793690672288 3loss/SRqqt2/assignment_loss‚0.45552342565852527 epoch‚48 # _timestamp‚1.78435683222262e+09 _runtime‚16490.915900805 3loss/FBt1/assignment_loss‚5.785475942384924e-06 /loss/FRt2/detection_loss‚1.2074321643009451 1loss/SRqqt2/detection_loss‚0.5571583422645539 trainer/global_step‚19249FÊð:Œ 1loss/SRqqt1/detection_loss‚0.8718075464396414 /loss/FBt1/detection_loss‚0.2652494969687333 _step‚1587 3loss/SRqqt1/assignment_loss‚0.37291254826573816 3loss/FBt2/assignment_loss‚3.677217315788817e-06 /loss/FRt1/detection_loss‚0.5379895444188305 0loss/FBt2/detection_loss‚0.34197137067061945 %loss/total_loss‚6.823807794532574 -loss/FRt1/assignment_loss‚1.4812745237845 0loss/FRt2/assignment_loss‚0.7324793690672288 3loss/SRqqt2/assignment_loss‚0.45552342565852527 epoch‚48 # _timestamp‚1.78435683222262e+09 _runtime‚16490.915900805 3loss/FBt1/assignment_loss‚5.785475942384924e-06 /loss/FRt2/detection_loss‚1.2074321643009451 1loss/SRqqt2/detection_loss‚0.5571583422645539 trainer/global_step‚19249³ ­@‹)œñ:‚(:‘ é¿ìÒ±Žà• gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚211.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.293000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÔâÛ:ò:‚(:¯ é¿ìÒÆ¿þ— disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.sda.out‚ 1724.9140625! disk.nvme0n1.in‚ 28326.9921875 network.sent‚ 3.0925766e+07$ disk./.usageGB‚585.6294250488281 disk.sda3.in‚ 417.107421875 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1554.37890625$ memory_percent‚97.76804447174072* disk./.usagePercent‚39.302082090843435 disk.sda3.out‚ 1554.37890625 disk.rbd0.in‚ 0.16015625( disk.nvme0c0n1.out‚35304.34619140625 disk.dm-0.in‚28312.95703125 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.drbd2165.in‚ 2.25390625# proc.memory.rssMB‚ 2809.07421875* proc.memory.percent‚17.145228385925293 cpu‚26.16232539204941 proc.cpu.threads‚17 disk.rbd1.out‚ 51.1611328125# disk.nvme0c0n1.in‚ 28326.9921875# disk.dm-0.out‚35304.34619140625 disk.sda1.in‚ 8.759765625& disk.nvme0n1.out‚35304.34619140625( proc.memory.availableMB‚ 365.68359375 disk.drbd2165.out‚0 network.recv‚ 4.378689e+06 disk.rbd0.out‚0 disk.sda.in‚541.0478515625èÙžó:‚(:“ ø¿ìÒâð¿“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?t_H*ô:‚(:Ÿ ø¿ìÒ¢¤ÿ– disk.sda2.out‚ 170.53515625$ disk.nvme0n1.out‚35308.693359375 network.recv‚ 4.383192e+06 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚28312.95703125! disk.dm-0.out‚35308.693359375 disk.sda.out‚ 1725.16015625! disk.nvme0n1.in‚ 28326.9921875 disk.sda3.out‚1554.625 disk.sda1.out‚0 disk.sda.in‚541.0478515625 disk.sda3.in‚ 417.107421875$ memory_percent‚97.76759147644043 cpu‚26.161894374183795' proc.memory.availableMB‚ 365.7578125 disk.dm-1.out‚1554.625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 28326.9921875 disk.rbd0.out‚0 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 51.1611328125 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625& disk.nvme0c0n1.out‚35308.693359375 disk.rbd0.in‚ 0.16015625# proc.memory.rssMB‚ 2809.07421875$ disk./.usageGB‚585.6305618286133* proc.memory.percent‚17.145228385925293 network.sent‚ 3.0988334e+07) disk./.usagePercent‚39.30215838108864¶ ÆB“õ:‚(:ˆ ‡ÀìÒ£”Í” gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{4¡§9ö:‚(:® ‡ÀìÒÒòÔ• disk.rbd0.in‚ 0.16015625 disk.sda2.out‚ 170.53515625 disk.sda1.out‚0 disk.dm-0.in‚ 28422.9765625 network.sent‚ 3.0993938e+07# disk./.usageGB‚585.622745513916 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1555.12890625$ disk.nvme0c0n1.in‚28437.01171875* proc.memory.percent‚17.145228385925293 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35315.66650390625 proc.cpu.threads‚17" disk.nvme0n1.in‚28437.01171875 disk.sda3.out‚ 1555.12890625 cpu‚26.16149542807055 network.recv‚ 4.386456e+06) disk./.usagePercent‚39.30163382165098 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875 disk.dm-1.in‚ 233.44140625( proc.memory.availableMB‚ 365.87890625& disk.nvme0n1.out‚35315.66650390625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0# disk.dm-0.out‚35315.66650390625$ memory_percent‚97.76685237884521# proc.memory.rssMB‚ 2809.07421875 disk.rbd1.out‚ 51.2236328125 disk.sda.out‚ 1725.6640625¤Ù+4÷:/ _wandbruntime‚16530  _runtime‚16530ªfˆNËø:Å $ _timestamp‚1.784356871501485e+09 _runtime‚16530.195042182 _step‚1588 ' lr-AdamW/pg1‚1.4158832715056842e-06 ' lr-AdamW/pg2‚1.4158832715056842e-06 trainer/global_step‚19299Ç –ºÐù:Ê $ _timestamp‚1.784356871501485e+09 _runtime‚16530.195042182 _step‚1588 ' lr-AdamW/pg1‚1.4158832715056842e-06 ' lr-AdamW/pg2‚1.4158832715056842e-06 trainer/global_step‚19299´ $kN®4ú:/ _wandbruntime‚16530  _runtime‚16530~z@€(û:€P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75298776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33805338 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41493189 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2919307085221888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153093428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843568715026271e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2954827502018560 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2934406895058944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178790783 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33805341 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666704836945920 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939696169580670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178800760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646062780647424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1589 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153093430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20420589920256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20420589920256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20389441948032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚19299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331884211 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153093430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646062799259648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20641949109248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954827143640064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666704729756672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153093428 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20420248581120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331894190 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2934406912098304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2954827502018560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178800760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ûsó©ªü:…P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33805341 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666704836945920 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939696169580670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178800760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646062780647424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1589 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153093430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20420589920256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20420589920256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20389441948032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚19299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331884211 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153093430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646062799259648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20641949109248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954827143640064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666704729756672 \ODeviceS∿ja tatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153093428 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20420248581120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331894190 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2934406912098304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2954827502018560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178800760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75298776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33805338 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41493189 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2919307085221888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153093428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843568715026271e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2954827502018560 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2934406895058944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178790783µ »eµ4ý:/ _wandbruntime‚16530  _runtime‚16530Eò\}'þ:‹N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20390453001866 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331900895 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20421261208064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178809826 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2954975755682304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2954975397296640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75302293 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚646096209405952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2934554153127936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331900895 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20643064081920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2934554136088576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178809826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41495468 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20390112451864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚16530.810708998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20421261208064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178799835 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20421602554368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2919453595893760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2934554153127936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33807086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153101062 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178799835 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1590 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331910888 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2939843691306264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33807083 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331910888 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚19299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153101062 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153101060 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2919453578854400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2954975755682304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20642977609216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666739292100096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20421602554368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666739187015168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843568721169744e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331910888 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2939843691306264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33807083 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331910888 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚19299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153101062 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153101060 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2919453578854400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2954975755682304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20642977609216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666739292100096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20421602554368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666739187015168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843568721169744e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2954975755682304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2954975397296640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75302293 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚646096209405952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2934554153127936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331900895 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20643064081920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2934554136088576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178809826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41495468 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20390112451864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚16530.810708998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20421261208064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178799835 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20421602554368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2919453595893760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2934554153127936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33807086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153101062¶ Žýë4€;/ _wandbruntime‚16530  _runtime‚16530g—µ ;… /loss/FRt1/assignment_loss‚1.385795745465308 0loss/FRt2/assignment_loss‚0.7074760378849122 2loss/SRqqt2/assignment_loss‚0.4384542038441244 /loss/FBt1/detection_loss‚0.4515924959643079 .loss/FBt2/detection_loss‚0.412952760577635 2loss/SRqqt1/assignment_loss‚0.3768776726938516 3loss/FBt1/assignment_loss‚1.908687416601091e-05 .loss/FRt2/detection_loss‚1.227447131541574 1loss/SRqqt2/detection_loss‚0.5420482700913757 % _timestamp‚1.7843568721201262e+09 _runtime‚16530.813408354 2loss/FBt2/assignment_loss‚0.002484973977098302 /loss/FRt1/detection_loss‚0.5323195058642176 trainer/global_step‚19299 _step‚1591 1loss/SRqqt1/detection_loss‚0.8676706453128465 %loss/total_loss‚6.945138530091418 epoch‚48ß=Ùu‚;Š 2loss/FBt2/assignment_loss‚0.002484973977098302 /loss/FRt1/detection_loss‚0.5323195058642176 trainer/global_step‚19299 _step‚1591 1loss/SRqqt1/detection_loss‚0.8676706453128465 %loss/total_loss‚6.945138530091418 epoch‚48 /loss/FRt1/assignment_loss‚1.385795745465308 0loss/FRt2/assignment_loss‚0.7074760378849122 2loss/SRqqt2/assignment_loss‚0.4384542038441244 /loss/FBt1/detection_loss‚0.4515924959643079 .loss/FBt2/detection_loss‚0.412952760577635 2loss/SRqqt1/assignment_loss‚0.3768776726938516 3loss/FBt1/assignment_loss‚1.908687416601091e-05 .loss/FRt2/detection_loss‚1.227447131541574 1loss/SRqqt2/detection_loss‚0.5420482700913757 % _timestamp‚1.7843568721201262e+09 _runtime‚16530.813408354· îr&’ƒ;‚(:‡ –ÀìÒï­£“ gpu.0.gpu‚16.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¥![6„;‚(:« –ÀìÒìÒÞ•$ disk./.usageGB‚585.6306266784668 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚28423.41015625! disk.nvme0n1.in‚ 28437.4453125( proc.memory.availableMB‚ 365.84765625# proc.memory.rssMB‚ 2809.07421875% disk.nvme0n1.out‚35318.3681640625 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1555.4921875 disk.sda.out‚ 1726.02734375# memory_percent‚97.7670431137085 cpu‚26.161092581747738 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 51.2236328125 disk.sda1.in‚ 8.759765625 disk.drbd2165.out‚0" disk.dm-0.out‚35318.3681640625 disk.sda.in‚541.0478515625* disk./.usagePercent‚39.302162733216726 disk.dm-1.in‚ 233.44140625 network.sent‚ 3.0998446e+07 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 disk.sda3.out‚ 1555.4921875 disk.sda1.out‚0' disk.nvme0c0n1.out‚35318.3681640625 proc.cpu.threads‚17 disk.sda2.in‚ 82.544921875# disk.nvme0c0n1.in‚ 28437.4453125* proc.memory.percent‚17.145228385925293 network.recv‚ 4.389875e+06YÝ3“…;‚(:ˆ ¥ÀìÒ¹ŠÎ“ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/c$=2†;‚(:§ ¥ÀìÒ¼·ó• network.recv‚ 4.394446e+06# disk.dm-0.out‚35323.20166015625 disk.rbd0.in‚ 0.16015625# proc.memory.rssMB‚ 2809.07421875 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625( disk.nvme0c0n1.out‚35323.20166015625 disk.sda.out‚ 1726.31640625 disk.sda3.out‚ 1555.78125* proc.memory.percent‚17.145228385925293$ disk./.usageGB‚585.6305999755859 disk.sda.in‚541.0478515625 cpu‚26.160690199963767 disk.dm-1.in‚ 233.44140625 disk.rbd1.out‚ 51.2861328125 disk.drbd2165.out‚0& disk.nvme0n1.out‚35323.20166015625 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 3.1062028e+07* disk./.usagePercent‚39.302160941163976# disk.nvme0c0n1.in‚ 28437.4453125 disk.dm-0.in‚28423.41015625 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.dm-1.out‚ 1555.78125 disk.drbd2165.in‚ 2.25390625! disk.nvme0n1.in‚ 28437.4453125 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875# memory_percent‚97.7670669555664% proc.memory.availableMB‚ 365.84375°“È4‡;/ _wandbruntime‚16570  _runtime‚16570ç Îˈ;Å ' lr-AdamW/pg1‚1.2212828508446455e-06 ' lr-AdamW/pg2‚1.2212828508446455e-06 trainer/global_step‚19349 $ _timestamp‚1.784356911549737e+09 _runtime‚16570.243297376 _step‚1592G î Љ;Ê ' lr-AdamW/pg1‚1.2212828508446455e-06 ' lr-AdamW/pg2‚1.2212828508446455e-06 trainer/global_step‚19349 $ _timestamp‚1.784356911549737e+09 _runtime‚16570.243297376 _step‚1592¸ 0!~{4Š;/ _wandbruntime‚16570  _runtime‚16570X!õ‹;€P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153475030 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41597673 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332730242 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20693138616320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚16570.24452828 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2962239890623488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179255212 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2941768963579904 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179245235 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2941768946540544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2962239890623488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2941768963579904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647734237186048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2962240249001984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.f*X÷—reed‚2941768946540544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20439717885050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20471285422080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153475030 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2947072677250428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470944082944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75490264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668427357190144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668427464379392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153475028 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20440058434940 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179255212 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20470944082944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2926632601776128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2962240249001984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33892591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41597394 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332720263 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153475028 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179245235 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843569115507984e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2926632618815488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2962239890623488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179255212 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2941768963579904 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179245235 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2941768946540544 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2962239890623488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2941768963579904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647734237186048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2962240249001984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2941768946540544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20439717885050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20471285422080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153475030 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2947072677250428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470944082944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75490264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668427357190144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668427464379392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153475028 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20440058434940 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179255212 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20470944082944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2926632601776128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2962240249001984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33892591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41597394 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332720263 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153475028 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179245235 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843569115507984e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2926632618815488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153482660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75493771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2926779129487360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179254287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20694168441344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33894336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332746940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647767647332352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332746940 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2941916187570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2962388502665728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647767665944576 _runtime‚16570.85794624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2941916187570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41599683 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33894333 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472298056192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2947219841386772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179254287 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2962388144280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20471956709888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚19349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 # _timestamp‚1.78435691216421e+09 _step‚1594 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2926779112448000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2941916204609536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153482662 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153482660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332736947 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179264278 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20471956709888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0æðÀ;N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179254287 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2962388144280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20471956709888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚19349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 # _timestamp‚1.78435691216421e+09 _step‚1594 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2926779112448000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2941916204609536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153482662 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153482660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332736947 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179264278 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20471956709888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179264278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20694250719744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2962388502665728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚668461916664320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153482660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75493771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2926779129487360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179254287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20694168441344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33894336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.onWz(Œ” _train_batch_end/allocation.all.allocated‚ 332746940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647767647332352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332746940 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2941916187570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2962388502665728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647767665944576 _runtime‚16570.85794624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2941916187570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41599683 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33894333 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472298056192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2947219841386772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269º qBÄÆ4;/ _wandbruntime‚16570  _runtime‚16570&$v‘;Š 1loss/SRqqt1/detection_loss‚0.7670348689534754 1loss/SRqqt2/detection_loss‚0.5876625545310422 /loss/FBt2/detection_loss‚0.3424748802753835 0loss/FBt1/detection_loss‚0.39880234246902546 trainer/global_step‚19349 _runtime‚16570.860953338 %loss/total_loss‚6.834462040903258 epoch‚48 4loss/FBt1/assignment_loss‚4.6516358935377866e-05 /loss/FRt1/detection_loss‚0.5093942998294565 /loss/FRt2/detection_loss‚1.1704568699120441 $ _timestamp‚1.784356912167653e+09 0loss/FRt2/assignment_loss‚0.7169149086470865 2loss/SRqqt1/assignment_loss‚0.4421719574492269 3loss/FBt2/assignment_loss‚0.0009322618045743737 _step‚1595 0loss/FRt1/assignment_loss‚1.5122324427795013 2loss/SRqqt2/assignment_loss‚0.3863381378935064—1Qà’; %loss/total_loss‚6.834462040903258 epoch‚48 4loss/FBt1/assignment_loss‚4.6516358935377866e-05 /loss/FRt1/detection_loss‚0.5093942998294565 /loss/FRt2/detection_loss‚1.1704568699120441 $ _timestamp‚1.784356912167653e+09 0loss/FRt2/assignment_loss‚0.7169149086470865 2loss/SRqqt1/assignment_loss‚0.4421719574492269 3loss/FBt2/assignment_loss‚0.0009322618045743737 _step‚1595 0loss/FRt1/assignment_loss‚1.5122324427795013 2loss/SRqqt2/assignment_loss‚0.3863381378935064 1loss/SRqqt1/detection_loss‚0.7670348689534754 1loss/SRqqt2/detection_loss‚0.5876625545310422 /loss/FBt2/detection_loss‚0.3424748802753835 0loss/FBt1/detection_loss‚0.39880234246902546 trainer/global_step‚19349 _runtime‚16570.860953338» q˜~ž“;‚(:“ ´ÀìÒ认• gpu.0.gpu‚84.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚241.365' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.272999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥êÕª3”;‚(:¨ ´ÀìÒú亗* proc.memory.percent‚17.145228385925293 network.sent‚ 3.1066417e+07* disk./.usagePercent‚39.302171437472886 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 28437.4453125 disk.dm-0.in‚28423.41015625 network.recv‚ 4.397168e+06 disk.drbd2165.out‚0# memory_percent‚97.7701187133789 disk.sda1.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚35326.3173828125 disk.sda1.in‚ 8.759765625" disk.dm-0.out‚35326.3173828125 disk.sda.out‚ 1726.9921875# proc.memory.rssMB‚ 2809.07421875 cpu‚26.16029897902221 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 28437.4453125 disk.rbd1.out‚ 51.2861328125 disk.rbd0.in‚ 0.16015625' disk.nvme0c0n1.out‚35326.3173828125 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1556.45703125% proc.memory.availableMB‚ 365.34375$ disk./.usageGB‚585.6307563781738 disk.sda3.out‚ 1556.45703125 disk.rbd1.in‚ 2.1796875 disk.drbd2165.in‚ 2.25390625¦5qБ•;‚(:† ÃÀìÒòÔª“ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â^$2–;‚(:§ ÃÀìÒáÆΕ disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 28437.4609375$ memory_percent‚97.76968955993652$ disk./.usageGB‚585.6308097839355 disk.drbd2165.out‚0 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 28437.4609375 disk.sda1.in‚ 8.759765625 disk.dm-0.in‚28423.42578125! disk.dm-0.out‚35329.064453125 disk.sda2.out‚ 170.53515625$ disk.nvme0n1.out‚35329.064453125* disk./.usagePercent‚39.302175021578364 disk.dm-1.in‚ 233.44140625& disk.nvme0c0n1.out‚35329.064453125 disk.sda.in‚541.0478515625 disk.rbd1.in‚ 2.1796875 network.recv‚ 4.401343e+06 disk.sda3.in‚ 417.107421875# proc.memory.rssMB‚ 2809.07421875 cpu‚26.15987453967088 proc.cpu.threads‚17 disk.dm-1.out‚ 1557.01171875 disk.sda.out‚ 1727.546875 disk.rbd0.out‚0' proc.memory.availableMB‚ 365.4140625 network.sent‚ 3.1128728e+07 disk.rbd0.in‚ 0.16015625 disk.sda3.out‚ 1557.01171875 disk.rbd1.out‚ 51.3486328125* proc.memory.percent‚17.145228385925293Õks“—;‚(:ˆ ÒÀìҨ⧓ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÉ3˜;‚(:¨ ÒÀìÒî“á• disk.sda2.out‚ 170.53515625* proc.memory.percent‚17.145228385925293" disk.nvme0n1.in‚28555.44140625 disk.rbd1.out‚ 51.3486328125 disk.rbd0.in‚ 0.16015625 disk.drbd2165.in‚ 2.25390625 disk.drbd2165.out‚0 disk.dm-0.in‚ 28541.40625 network.sent‚ 3.1134414e+07) disk./.usagePercent‚39.30217706963864 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625# proc.memory.rssMB‚ 2809.07421875 proc.cpu.threads‚17$ disk./.usageGB‚585.6308403015137 disk.sda.out‚ 1727.859375 disk.sda3.out‚ 1557.32421875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚35333.6044921875 network.recv‚ 4.404408e+06 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.sda.in‚541.0478515625% disk.nvme0n1.out‚35333.6044921875 disk.sda1.in‚ 8.759765625$ disk.nvme0c0n1.in‚28555.44140625$ memory_percent‚97.76968955993652 cpu‚26.15947438236562 disk.dm-1.out‚ 1557.32421875" disk.dm-0.out‚35333.6044921875' proc.memory.availableMB‚ 365.4140625]H°»4™;/ _wandbruntime‚16610  _runtime‚16610]¶Ù¿Ëš;Å trainer/global_step‚19399 $ _timestamp‚1.784356951391442e+09 _runtime‚16610.085055777 _step‚1596 ' lr-AdamW/pg1‚1.0410256758973856e-06 ' lr-AdamW/pg2‚1.0410256758973856e-06”Ëc>Л;Ê ' lr-AdamW/pg1‚1.0410256758973856e-06 ' lr-AdamW/pg2‚1.0410256758973856e-06 trainer/global_step‚19399 $ _timestamp‚1.784356951391442e+09 _runtime‚16610.085055777 _step‚1596¼ y—ï4œ;/ _wandbruntime‚16610  _runtime‚16610š¨æÊ(;€P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41701864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179709664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚670150028912128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2933958135369728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179709664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2933958152409088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2949131015061504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75681702 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚16610.08641824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333566294 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843569513926861e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20744353799680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20521980923904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41702139 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969652637606912 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2949131015061504 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2949130998022144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚19399 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969652637606912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969652995985408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153856628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649405675112448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333556315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20490334371958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969652995985408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20490674921848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153856630 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2949130998022144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20521980923904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153856628 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2954448827330936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179699687 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153856630 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2949130998022144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20521980923904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153856628 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2954448827330936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179699687 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2949131015061504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75681702 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚16610.08641824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333566294 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843569513926861e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20744353799680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20521980923904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41702139 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969652637606912 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2949131015061504 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2949130998022144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚19399 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969652637606912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969652995985408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153856628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649405675112448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333556315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20490334371958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969652995985408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20490674921848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179708739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179718730 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333572999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20522652211712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670184362198016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153864262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20522652211712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚16610.700696815 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33981586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522993558016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333582992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75685446 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2954595991467280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2934104646041600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2969800891263488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179718730 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20522993558016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153864260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20491345425680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2954596349056642 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 333572999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33981583 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41703863 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153864260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2934104663080960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20491685975682 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2949278256091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20745276939264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75685727 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269Oë°)'¡;N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20522993558016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153864260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20491345425680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2954596349056642 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 333572999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33981583 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41703863 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153864260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2934104663080960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20491685975682 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2949278256091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20745276939264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75685727 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2969801249649152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179708739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179718730 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333572999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20522652211712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670184362198016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153864262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20522652211712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚16610.700696815 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33981586 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522993558016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333582992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75685446 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2954595991467280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2934104646041600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2969800891263488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179718730 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278¾ yÛÜç4¢;/ _wandbruntime‚16610  _runtime‚16610ç7)Q£;‹ 3loss/SRqqt2/assignment_loss‚0.41066461671087734 3loss/FBt1/assignment_loss‚0.0037975167253524555 3loss/FBt2/assignment_loss‚0.0047572337464523275 /loss/FRt1/detection_loss‚0.5417643485850823 /loss/FRt2/detection_loss‚1.2156194681005779 1loss/SRqqt1/detection_loss‚0.7813913324155853 1loss/SRqqt2/detection_loss‚0.5231079532539908 0loss/FBt2/detection_loss‚0.29133807612839524 3loss/SRqqt1/assignment_loss‚0.41956859013254294 _runtime‚16610.702913109 _step‚1599 0loss/FRt1/assignment_loss‚1.4998257730716409 0loss/FBt1/detection_loss‚0.44653859498783166 $loss/total_loss‚6.79456042665809 trainer/global_step‚19399 0loss/FRt2/assignment_loss‚0.6561869227997611 epoch‚48 $ _timestamp‚1.784356952009627e+09q¦ÑT¤; 0loss/FRt1/assignment_loss‚1.4998257730716409 0loss/FBt1/detection_loss‚0.44653859498783166 $loss/total_loss‚6.79456042665809 trainer/global_step‚19399 0loss/FRt2/assignment_loss‚0.6561869227997611 epoch‚48 $ _timestamp‚1.784356952009627e+09 3loss/SRqqt2/assignment_loss‚0.41066461671087734 3loss/FBt1/assignment_loss‚0.0037975167253524555 3loss/FBt2/assignment_loss‚0.0047572337464523275 /loss/FRt1/detection_loss‚0.5417643485850823 /loss/FRt2/detection_loss‚1.2156194681005779 1loss/SRqqt1/detection_loss‚0.7813913324155853 1loss/SRqqt2/detection_loss‚0.5231079532539908 0loss/FBt2/detection_loss‚0.29133807612839524 3loss/SRqqt1/assignment_loss‚0.41956859013254294 _runtime‚16610.702913109 _step‚1599¿ b€-›¥;‚(: áÀìÒñŒ“ gpu.0.gpu‚5.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IP8ó9¦;‚(:® áÀìҲߵ– disk.sda.in‚541.0478515625$ memory_percent‚97.77014255523682$ disk./.usageGB‚585.6308326721191 disk.sda1.in‚ 8.759765625$ disk.nvme0c0n1.in‚28555.44140625 disk.dm-0.in‚ 28541.40625 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625& disk.nvme0n1.out‚35336.48193359375 cpu‚26.159073281107908( disk.nvme0c0n1.out‚35336.48193359375# disk.dm-0.out‚35336.48193359375 disk.sda.out‚ 1728.3515625 disk.sda2.in‚ 82.544921875 disk.rbd1.in‚ 2.1796875) disk./.usagePercent‚39.30217655762357 disk.rbd1.out‚ 51.4736328125 disk.rbd0.out‚0 disk.dm-1.out‚ 1557.81640625 disk.sda3.in‚ 417.107421875# proc.memory.rssMB‚ 2809.07421875* proc.memory.percent‚17.145228385925293 network.sent‚ 3.1139558e+07 network.recv‚ 4.407875e+06 disk.dm-1.in‚ 233.44140625" disk.nvme0n1.in‚28555.44140625 disk.sda3.out‚ 1557.81640625( proc.memory.availableMB‚ 365.33984375 disk.drbd2165.out‚0°Ââv“§;‚(:ˆ ðÀìÒñóä“ gpu.0.gpu‚37.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F†˜3¨;‚(:¨ ðÀìÒËëß• disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625 cpu‚26.158692225689233* disk./.usagePercent‚39.302170413442745 disk.rbd1.in‚ 2.1796875% disk.nvme0n1.out‚35341.3525390625* proc.memory.percent‚17.145228385925293 network.recv‚ 4.412229e+06 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0 disk.sda1.in‚ 8.759765625' disk.nvme0c0n1.out‚35341.3525390625" disk.dm-0.out‚35341.3525390625" disk.nvme0n1.in‚28555.44140625& proc.memory.availableMB‚ 365.359375$ disk./.usageGB‚585.6307411193848 disk.rbd1.out‚ 51.4736328125 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.dm-0.in‚ 28541.40625 disk.sda2.in‚ 82.544921875 disk.sda3.out‚ 1557.99609375 proc.cpu.threads‚17 disk.dm-1.out‚ 1557.99609375$ disk.nvme0c0n1.in‚28555.44140625 disk.sda.out‚ 1728.53125 disk.sda3.in‚ 417.107421875 network.sent‚ 3.1202665e+07 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625$ memory_percent‚97.77002334594727# proc.memory.rssMB‚ 2809.07421875ÿž6ï©;‚(:’ ÿÀìÒ银“ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¦9ª;‚(:® ÿÀìÒü嶕* disk./.usagePercent‚39.302171437472886# disk.dm-0.out‚35344.42529296875" disk.nvme0n1.in‚28555.44140625& disk.nvme0n1.out‚35344.42529296875 proc.cpu.threads‚17 disk.dm-0.in‚ 28541.40625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35344.42529296875( proc.memory.availableMB‚ 365.37109375 disk.rbd1.out‚ 51.5361328125 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1558.5078125 disk.drbd2165.out‚0 cpu‚26.158250793011096 network.sent‚ 3.1207189e+07 disk.drbd2165.in‚ 2.25390625 disk.sda3.out‚ 1558.5078125 network.recv‚ 4.415481e+06 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875 disk.sda1.out‚0* proc.memory.percent‚17.145228385925293 disk.sda2.in‚ 82.544921875$ memory_percent‚97.76995182037354 disk.sda.out‚ 1729.04296875$ disk.nvme0c0n1.in‚28555.44140625# proc.memory.rssMB‚ 2809.07421875$ disk./.usageGB‚585.6307563781738 disk.sda.in‚541.0478515625 disk.rbd1.in‚ 2.17968753ýmò4«;/ _wandbruntime‚16650  _runtime‚16650kcBDʬ;Ä _runtime‚16650.109261886 _step‚1600 & lr-AdamW/pg1‚8.751233242453668e-07 & lr-AdamW/pg2‚8.751233242453668e-07 trainer/global_step‚19449 % _timestamp‚1.7843569914155145e+09ˆŸÏ­;É trainer/global_step‚19449 % _timestamp‚1.7843569914155145e+09 _runtime‚16650.109261886 _step‚1600 & lr-AdamW/pg1‚8.751233242453668e-07 & lr-AdamW/pg2‚8.751233242453668e-07À nÌJ¦4®;/ _wandbruntime‚16650  _runtime‚16650Æ$ߧ¯;€P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154238228 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2956493049503744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651077113038848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2956493049503744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180154139 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75873051 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2961824619822194 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2941283668963328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34067091 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180164116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977065384590336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180164116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20540950858866 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154238230 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180154139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41805706 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20572335086592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2941283686002688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2956493066543104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843569914168067e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334392367 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20572335086592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180154139 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75873051 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2961824619822194 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2941283668963328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34067091 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180164116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977065384590336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180164116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20540950858866 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154238230 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180154139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41805706 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20572335086592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2941283686002688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2956493066543104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843569914168067e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334392367 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20572335086592 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2956640290533376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75876870 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75876605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20541961912588 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334409051 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977213996632576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180163191 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2941430196674560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180173182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671906880329216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1602 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20573689059840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334419044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚19449 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977213638246912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2956640307572736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154245862 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651110541797376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2941430179635200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34068836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2961972499137150 % _timestamp‚1.7843569920312595e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651110523185152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334419044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20573689059840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154245860 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20573347713536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20796357144064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180173182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2977213996632576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41808034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2961972141547788 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚16650.724969648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2977213996632576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41808034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2961972141547788 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚16650.724969648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2956640290533376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75876870 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75876605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20541961912588 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334409051 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977213996632576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180163191 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2941430196674560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180173182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671906880329216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1602 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20573689059840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334419044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚19449 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977213638246912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2956640307572736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154245862 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651110541797376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2941430179635200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34068836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2961972499137150 % _timestamp‚1.7843569920312595e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651110523185152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334419044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20573689059840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 /¯ð4´;/ _wandbruntime‚16650  _runtime‚16650åQH׿µ;‰ 4loss/FBt1/assignment_loss‚1.9232629452628053e-05 /loss/FBt2/detection_loss‚0.4265968997331371 %loss/total_loss‚6.803860696339981 epoch‚48 trainer/global_step‚194FP üP49 1loss/SRqqt1/detection_loss‚0.8504083305177791 _runtime‚16650.727135197 /loss/FRt2/assignment_loss‚0.752005710785568 2loss/SRqqt1/assignment_loss‚0.5226793369469549 2loss/SRqqt2/assignment_loss‚0.4039879777934178 2loss/FBt2/assignment_loss‚0.007940112756667794 1loss/SRqqt2/detection_loss‚0.5212590342634026 % _timestamp‚1.7843569920338538e+09 0loss/FRt1/assignment_loss‚1.4234821577395858 /loss/FRt1/detection_loss‚0.5117799679568074 /loss/FRt2/detection_loss‚1.1463463960678235 0loss/FBt1/detection_loss‚0.23735553914938415 _step‚1603Ø„y¶;Ž 1loss/SRqqt1/detection_loss‚0.8504083305177791 _runtime‚16650.727135197 /loss/FRt2/assignment_loss‚0.752005710785568 2loss/SRqqt1/assignment_loss‚0.5226793369469549 2loss/SRqqt2/assignment_loss‚0.4039879777934178 2loss/FBt2/assignment_loss‚0.007940112756667794 1loss/SRqqt2/detection_loss‚0.5212590342634026 % _timestamp‚1.7843569920338538e+09 0loss/FRt1/assignment_loss‚1.4234821577395858 /loss/FRt1/detection_loss‚0.5117799679568074 /loss/FRt2/detection_loss‚1.1463463960678235 0loss/FBt1/detection_loss‚0.23735553914938415 _step‚1603 4loss/FBt1/assignment_loss‚1.9232629452628053e-05 /loss/FBt2/detection_loss‚0.4265968997331371 %loss/total_loss‚6.803860696339981 epoch‚48 trainer/global_step‚19449Ã Ý X5ž·;‚(:“ ŽÁìÒ‚†¿“ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÅÜ7¸;‚(:¬ ŽÁìÒÂË¿— disk.sda.in‚541.0478515625 disk.sda.out‚ 1729.5078125 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 28679.359375$ memory_percent‚97.76921272277832 network.sent‚ 3.1212774e+07 disk.sda1.out‚0 disk.rbd1.out‚ 51.5361328125* proc.memory.percent‚17.145228385925293 proc.cpu.threads‚17 network.recv‚ 4.418689e+06 disk.drbd2165.in‚ 2.25390625 disk.dm-0.in‚28665.32421875# disk.dm-0.out‚35349.97119140625' proc.memory.availableMB‚ 365.4921875# proc.memory.rssMB‚ 2809.07421875 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625" disk.nvme0c0n1.in‚ 28679.359375( disk.nvme0c0n1.out‚35349.97119140625 disk.sda2.in‚ 82.544921875& disk.nvme0n1.out‚35349.97119140625 disk.rbd1.in‚ 2.1796875* disk./.usagePercent‚39.302166573329735$ disk./.usageGB‚585.6306838989258 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 1558.97265625 cpu‚26.15786221470559 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1558.97265625 disk.sda3.in‚ 417.107421875§Ê;k’¹;‚(:‡ ÁìÒ°¹“ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚252.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£3^m6º;‚(:« ÁìÒƒ”‚– disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0 disk.dm-1.out‚ 1559.19921875 disk.sda.in‚541.0478515625 disk.rbd1.in‚ 2.1796875 cpu‚26.157485919051656$ disk./.usageGB‚585.6306190490723 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0# disk.dm-0.out‚35352.83251953125& disk.nvme0n1.out‚35352.83251953125 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1559.19921875* proc.memory.percent‚17.145228385925293 disk.rbd0.out‚0 disk.dm-0.in‚28665.32421875 disk.sda2.in‚ 82.544921875 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625" disk.nvme0c0n1.in‚ 28679.359375( disk.nvme0c0n1.out‚35352.83251953125 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 28679.359375 disk.drbd2165.in‚ 2.25390625 disk.rbd1.out‚ 51.5986328125# proc.memory.rssMB‚ 2809.07421875 network.sent‚ 3.1276087e+07 network.recv‚ 4.423245e+06* disk./.usagePercent‚39.302162221201655 disk.sda.out‚ 1729.734375$ memory_percent‚97.76926040649414& proc.memory.availableMB‚ 365.484375Äή4»;/ _wandbruntime‚16690  _runtime‚16690TmÈʼ;Ä % _timestamp‚1.7843570316587791e+09 _runtime‚16690.352360756 _step‚1604 & lr-AdamW/pg1‚7.235864514866237e-07 & lr-AdamW/pg2‚7.235864514866237e-07 trainer/global_step‚19499ä=ÑTϽ;É & lr-AdamW/pg1‚7.235864514866237e-07 & lr-AdamW/pg2‚7.235864514866237e-07 trainer/global_step‚19499 % _timestamp‚1.7843570316587791e+09 _runtime‚16690.352360756 _step‚1604Ä ™7éú4¾;/ _wandbruntime‚16690  _runtime‚16690ˆ}êþ(¿;€P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20623030588416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76063971 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20846356665344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚16690.35351773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20623030588416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335228419 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984478131573760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652748550965248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2963855100985344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2969201127491952 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2948609202556928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20623371927552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚673595006693376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180608591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2969200769902702 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20591907895664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34154341 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76064238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2963855118024704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376†¯*Ì (À;…P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚19499 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2984478489952256 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34154341 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76064238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2963855118024704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20623030588416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76063971 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20846356665344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚16690.35351773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20623030588416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335228419 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984478131573760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652748550965248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2963855100985344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2969201127491952 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2948609202556928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20623371927552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚673595006693376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180608591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2969200769902702 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20591907895664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180627634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154627460 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335255096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20592578399496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843570322737675e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154627460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2964002342014976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2964002359054336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41911659 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20624043215360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652781961111552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2969348649217658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2948755730268160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76067742 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20624384561664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41911944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673629338446336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2964002342014976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180617643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34156086 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34156083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2948755713228800 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2984626385230336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1606 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_e­ÝẠnd/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19499 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335255096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673629449822720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154627462ƒKØÀ'Ã;N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652781961111552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2969348649217658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2948755730268160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76067742 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20624384561664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41911944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673629338446336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2964002342014976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180617643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34156086 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34156083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2948755713228800 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2984626385230336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚1606 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19499 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335255096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673629449822720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154627462 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20624043215360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180617643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2984626743616000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335245103 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2984626743616000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20592918949498 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76068030 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180627634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154627460 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335255096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20592578399496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843570322737675e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154627460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2964002342014976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2964002359054336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41911659 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20624043215360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101Æ ÂPk;4Ä;/ _wandbruntime‚16690  _runtime‚16690ÿ†_Å;‰ 3loss/SRqqt1/assignment_loss‚0.46861180897930116 4loss/FBt1/assignment_loss‚2.0686288363315405e-07 4loss/FBt2/assignment_loss‚4.5983544508026596e-05 1loss/SRqqt1/detection_loss‚0.7497290213000706 0loss/FRt1/assignment_loss‚1.4948082367363886 /loss/FBt2/detection_loss‚0.3434772276953584 0loss/FRt2/assignment_loss‚0.8202053467672034 2loss/SRqqt2/assignment_loss‚0.4834123451031606 epoch‚48 trainer/global_step‚19499 _step‚1607 /loss/FRt1/detection_loss‚0.5382924761852603 .loss/FRt2/detection_loss‚1.158431838099409 1loss/SRqqt2/detection_loss‚0.5816582347464654 .loss/FBt1/detection_loss‚0.521670113907449 %loss/total_loss‚7.160342839927458 % _timestamp‚1.7843570322765808e+09 _runtime‚16690.96987934÷q¶Æ;Ž 0loss/FRt1/assignment_loss‚1.4948082367363886 /loss/FBt2/detection_loss‚0.3434772276953584 0loss/FRt2/assignment_loss‚0.8202053467672034 2loss/SRqqt2/assignment_loss‚0.4834123451031606 epoch‚48 trainer/global_step‚19499 _step‚1607 /loss/FRt1/detection_loss‚0.5382924761852603 .loss/FRt2/detection_loss‚1.158431838099409 1loss/SRqqt2/detection_loss‚0.5816582347464654 .loss/FBt1/detection_loss‚0.521670113907449 %loss/total_loss‚7.160342839927458 % _timestamp‚1.7843570322765808e+09 _runtime‚16690.96987934 3loss/SRqqt1/assignment_loss‚0.46861180897930116 4loss/FBt1/assignment_loss‚2.0686288363315405e-07 4loss/FBt2/assignment_loss‚4.5983544508026596e-05 1loss/SRqqt1/detection_loss‚0.7497290213000706Ç rzƒ“Ç;‚(:ˆ ¬ÁìÒëا“ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ô4È;‚(:© ¬ÁìÒÅLJ–$ disk./.usageGB‚585.6306190490723 disk.dm-0.in‚28665.32421875 disk.sda1.out‚0' proc.memory.availableMB‚ 365.5078125 cpu‚26.15753886613458 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 28679.359375 disk.sda1.in‚ 8.759765625# proc.memory.rssMB‚ 2809.07421875* proc.memory.percent‚17.145228385925293 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625% disk.nvme0n1.out‚35357.5751953125$ memory_percent‚97.76911735534668 network.recv‚ 4.426356e+06 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.dm-1.out‚ 1559.50390625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 network.sent‚ 3.1281138e+07* disk./.usagePercent‚39.302162221201655 disk.dm-1.in‚ 233.44140625 disk.sda.out‚ 1730.0390625 disk.rbd1.out‚ 51.5986328125 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0" disk.dm-0.out‚35357.5751953125' disk.nvme0c0n1.out‚35357.5751953125 disk.sda.in‚541.0478515625 disk.nvme0n1.in‚ 28679.359375 disk.sda3.out‚ 1559.50390625 ù'òžÉ;‚(:“ »ÁìÒ݇á“ gpu.0.gpu‚68.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÍ ™1Ê;‚(:¦ »ÁìҠ艖* disk./.usagePercent‚39.301639965831804 disk.sda2.out‚ 170.53515625 disk.dm-1.in‚ 233.44140625( proc.memory.availableMB‚ 365.12890625 disk.dm-1.out‚ 1559.98046875$ memory_percent‚97.77143001556396 cpu‚26.157726219062145 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚35362.708984375 disk.dm-0.in‚28665.32421875 disk.sda.in‚541.0478515625! disk.dm-0.out‚35362.708984375$ disk.nvme0n1.out‚35362.708984375 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625* proc.memory.percent‚17.145228385925293 network.recv‚ 4.43066e+06" disk.nvme0c0n1.in‚ 28679.359375 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1559.98046875 disk.sda1.out‚0 disk.sda.out‚ 1730.515625 disk.rbd1.out‚ 51.6611328125 disk.rbd0.in‚ 0.16015625 proc.cpu.threads‚17 network.sent‚ 3.1344196e+07 disk.nvme0n1.in‚ 28679.359375 disk.rbd1.in‚ 2.1796875$ disk./.usageGB‚585.6228370666504 disk.drbd2165.in‚ 2.25390625# proc.memory.rssMB‚ 2809.074218754¼6“Ë;‚(:ˆ ÊÁìÒºÉÇ“ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FçB6Ì;‚(:« ÊÁìÒ¥ž¯–# proc.memory.rssMB‚ 2809.07421875 proc.cpu.threads‚17$ disk./.usageGB‚585.6229057312012 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1730.8515625 network.sent‚ 3.1349358e+07 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1560.31640625* proc.memory.percent‚17.145228385925293 disk.sda2.in‚ 82.544921875! disk.nvme0n1.in‚ 28807.4921875' proc.memory.availableMB‚ 365.1328125" disk.dm-0.out‚35369.6044921875 disk.rbd1.out‚ 51.6611328125 disk.dm-1.in‚ 233.44140625 cpu‚26.15791232914846 disk.sda1.in‚ 8.759765625# disk.nvme0c0n1.in‚ 28807.4921875 disk.sda2.out‚ 170.53515625% disk.nvme0n1.out‚35369.6044921875* disk./.usagePercent‚39.301644573967415 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1560.31640625$ memory_percent‚97.77140617370605 network.recv‚ 4.433775e+06 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625' disk.nvme0c0n1.out‚35369.6044921875 disk.dm-0.in‚28793.45703125 disk.sda.in‚541.0478515625e§Zá4Í;/ _wandbruntime‚16730  _runtime‚16730G$uîÊÎ;Ä & lr-AdamW/pg2‚5.864247905513426e-07 trainer/global_step‚19549 % _timestamp‚1.7843570721287878e+09 _runtime‚16730.822458287 _step‚1608 & lr-AdamW/pg1‚5.864247905513426e-07Šm½øÏÏ;É _runtime‚16730.822458287 _step‚1608 & lr-AdamW/pg1‚5.864247905513426e-07 & lr-AdamW/pg2‚5.864247905513426e-07 trainer/global_step‚19549 % _timestamp‚1.7843570721287878e+09È ËG4Ð;/ _wandbruntime‚16730  _runtime‚16730Ø=f(Ñ;P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675317368851456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155001428 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20642183832682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42013039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336074450 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2971217169506304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2991891236935680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675317482332160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚19549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7843570721299407e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2971217169506304 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155001428 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20642524382572 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336064471 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181063043 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155001430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20897493440512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34241591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2955934736150528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2991891236935680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2991890878557184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34241588 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚16730.823679412 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2955934753189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76254347 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181073020 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42012759 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20674067429376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20673726090240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181063043 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181073020 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2971217152466944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155001430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654419988891648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20897398572032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2976576919983210 _step‚1609 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20674067429376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2991890878557184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155001430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654419988891648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20897398572032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2976576919983210 _step‚1609 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20674067429376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2991890878557184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_ôÐ7Apool.peak‚479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20642183832682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42013039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336074450 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2971217169506304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2991891236935680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675317482332160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚19549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7843570721299407e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2971217169506304 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155001428 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20642524382572 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336064471 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181063043 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155001430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20897493440512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34241591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2955934736150528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2991891236935680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2991890878557184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34241588 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚16730.823679412 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2955934753189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76254347 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181073020 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42012759 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20674067429376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20673726090240É âµ$*4Ó;/ _wandbruntime‚16731  _runtime‚16731´Y³æ'Ô;ŒN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181082086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42014787 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2992039132213760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2976724799298166 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336091148 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181072095 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20898515525120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155009062 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675351933175296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76258120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155009060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20643194886404 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2976724441708804 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2992039490599424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336091148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1610•]¡G'Õ;‘N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20674738717184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34243333 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2956081263861760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2956081246822400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2992039490599424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20675080063488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34243336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336081155 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20643535436406 _runtime‚16731.437844811 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2971364410535936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654453417650176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 trainer/global_step‚19549 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843570727441103e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76258120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155009060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20643194886404 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2976724441708804 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2992039490599424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336091148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1610 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181082086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42014787 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2992039132213760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2976724799298166 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336091148 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181072095 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20898515525120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155009062 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675351933175296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20724762931200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155383028 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978579220987904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20724421592064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20724421592064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76445388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2999303625540608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42116295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978579220987904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336900523 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383030 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181527472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999303625540608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚656091408205824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181527472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚16771.423490987 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336910502 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181517495 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2963260286783488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20692800319590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383030 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34328838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2999303983919104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2999303983919104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677040003451392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656091426818048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843571127297475e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚656091408205824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181527472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚16771.423490987 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336910502 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181517495 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2963260286783488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20692800319590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383030 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34328838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2999303983919104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2999303983919104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677040003451392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656091426818048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843571127297475e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20724762931200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155383028 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978579220987904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20724421592064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20724421592064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76445388 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2999303625540608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42116295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978579220987904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336900523 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383030 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181527472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999303625540608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76448878 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚677074350560768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656124836964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚19599 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42118555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20725434219008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336927200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1614 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2999451879197184 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155390660 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2984100591789312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34330586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20949600069120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181526547 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20693811373312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2978726462017536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20725775565312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20725434219008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42118295 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20694151923314 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181526547 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155390662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181536538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚16772.036408555 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2984100949378674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20949513596416Ár¨(Õç;N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20949513596416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181536538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76448878 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚677074350560768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656124836964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚19599 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.oy1DÓA#n_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42118555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20725434219008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336927200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1614 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2999451879197184 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155390660 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2984100591789312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34330586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20949600069120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181526547 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20693811373312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2978726462017536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20725775565312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20725434219008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42118295 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20694151923314 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181526547 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155390662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181536538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚16772.036408555 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2984100949378674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Î ›£’¬4è;/ _wandbruntime‚16772  _runtime‚16772¬±Eé;‰ 3loss/FBt2/assignment_loss‚0.0001335793020617722 epoch‚48 _runtime‚16772.03977934 %loss/total_loss‚7.032349992769722 0loss/FRt1/assignment_loss‚1.4817934060383677 0loss/FRt2/assignment_loss‚0.7536691661092118 -loss/FRt1/detection_loss‚0.55528239560087 1loss/SRqqt1/detection_loss‚0.9360325629123674 1loss/SRqqt2/detection_loss‚0.5580085326598453 /loss/FBt2/detection_loss‚0.3448620314102183 % _timestamp‚1.7843571133464766e+09 2loss/SRqqt1/assignment_loss‚0.4228523828345019 3loss/SRqqt2/assignment_loss‚0.41568868490623484 4loss/FBt1/assignment_loss‚0.00010036175055184849 _step‚1615 /loss/FRt2/detection_loss‚1.2358112037024915 0loss/FBt1/detection_loss‚0.32811568554300014 trainer/global_step‚19599Ó$ø(ê;Ž trainer/global_step‚19599 3loss/FBt2/assignment_loss‚0.0001335793020617722 epoch‚48 _runtime‚16772.03977934 %loss/total_loss‚7.032349992769722 0loss/FRt1/assignment_loss‚1.4817934060383677 0loss/FRt2/assignment_loss‚0.7536691661092118 -loss/FRt1/detection_loss‚0.55528239560087 1loss/SRqqt1/detection_loss‚0.9360325629123674 1loss/SRqqt2/detection_loss‚0.5580085326598453 /loss/FBt2/detection_loss‚0.3448620314102183 % _timestamp‚1.7843571133464766e+09 2loss/SRqqt1/assignment_loss‚0.4228523828345019 3loss/SRqqt2/assignment_loss‚0.41568868490623484 4loss/FBt1/assignment_loss‚0.00010036175055184849 _step‚1615 /loss/FRt2/detection_loss‚1.2358112037024915 0loss/FBt1/detection_loss‚0.32811568554300014Ï |Ðæ_‘ë;‚(:† †ÂìÒ¡³€• gpu.0.gpu‚44.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îfÅ3ì;‚(:¨ †ÂìÒ»Õ­– disk.sda1.out‚0& disk.nvme0n1.out‚35412.41748046875$ memory_percent‚97.78375625610352 network.recv‚ 4.448489e+06 disk.dm-0.in‚28916.26171875# disk.dm-0.out‚35412.41748046875 disk.rbd1.out‚ 51.7861328125 disk.drbd2165.out‚0 disk.sda2.in‚ 82.544921875 disk.nvme0n1.in‚ 28930.296875" proc.memory.rssMB‚ 2812.0390625 disk.rbd1.in‚ 2.1796875 disk.drbd2165.in‚ 2.25390625 disk.sda.in‚541.0478515625 proc.cpu.threads‚17 disk.sda3.in‚ 417.107421875 network.sent‚ 3.1427116e+07$ disk./.usageGB‚585.6429138183594 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35412.41748046875) disk./.usagePercent‚39.30298733348451 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625" disk.nvme0c0n1.in‚ 28930.296875 disk.sda.out‚ 1738.0390625 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1567.50390625( proc.memory.percent‚17.1633243560791 disk.dm-1.out‚ 1567.50390625& proc.memory.availableMB‚ 363.109375 cpu‚26.161303953465563™£N§4í;/ _wandbruntime‚16790  _runtime‚167900%ìÿ‰=î;ƒ{ 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2131815254688263 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23621991276741028 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6378304362297058 _step‚1616 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09896577894687653 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31536394357681274 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28279343247413635 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7091246843338013 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7243047952651978 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04078979045152664 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1661490648984909 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.34687697887420654 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7870433330535889 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 *jet/accuracy_1_of_3‚0.9266890287399292 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28309330344200134 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27855268120765686 7validation_average_jet_accuracy‚0.37964314222335815 % _timestamp‚1.7843571313887463e+09 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13543260097503662 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7852444648742676 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.768031895160675 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23797576129436493 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28309330344200134 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08790280669927597 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_3_of_3‚0.03792091831564903 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2413233071565628 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051215432584285736 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23109839856624603 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8287604451179504 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5764477849006653 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397027969360352 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925015687942505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6524272561073303 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 particle/accuracy_5_of_5‚1 trainer/global_step‚19599 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6551302075386047 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_1_of_4‚0.9806751012802124 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27468886971473694 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2830818295478821 *jet/accuracy_2_of_2‚0.1260158121585846 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6537662148475647 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7750176191329956 /particle/accuracy_3_of_4‚0.9968254566192627 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6122134327888489 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7272027134895325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3767842650413513 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24848969280719757 +particle/specificity‚0.8985968828201294 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04559728130698204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6387445330619812 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2124873548746109 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1426972597837448 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5671265721321106 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6823356747627258 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782057166099548 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6285480260848999 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2492525428533554 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29360127449035645 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6681058406829834 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6576123237609863 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29627126455307007 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24590946733951569 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15205833315849304 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8332480192184448 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24797183275222778 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6048401594161987 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3634704351425171 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7843713164329529 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.5942925810813904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6907509565353394 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14929237961769104 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6830228567123413 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03633022680878639 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7337389588356018 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2938586175441742 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5112574696540833 +jet/accuracy_4_of_4‚0.01670061983168125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3382278084754944 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23804312944412231 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3295789062976837 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚16790.082954213 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7136529088020325 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5520079135894775 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 )loss/val_total_loss‚7.703847521586939 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7073653340339661 particle/accuracy_1_of_5‚1 epoch‚48 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6693097352981567 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03611055016517639 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5683405995368958 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6378304362297058 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7274854779243469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2220502644777298 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360731482505798 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_3_of_3‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09647992998361588 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6308538317680359 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6919464468955994 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9452828764915466 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957;JÖD¨ï;ˆ{ & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3767842650413513 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24848969280719757 +particle/specificity‚0.8985968828201294 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04559728130698204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6387445330619812 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2124873548746109 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1426972597837448 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5671265721321106 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6823356747627258 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782057166099548 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6285480260848999 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2492525428533554 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29360127449035645 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6681058406829834 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6576123237609863 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29627126455307007 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24590946733951569 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15205833315849304 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8332480192184448 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24797183275222778 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6048401594161987 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3634704351425171 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7843713164329529 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.5942925810813904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6907509565353394 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14929237961769104 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6830228567123413 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03633022680878639 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7337389588356018 *jet/accuracy_2_of_4‚0.7723941802978516 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2938586175441742 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1ín8 æ* @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5112574696540833 +jet/accuracy_4_of_4‚0.01670061983168125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3382278084754944 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23804312944412231 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3295789062976837 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚16790.082954213 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7136529088020325 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5520079135894775 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 )loss/val_total_loss‚7.703847521586939 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7073653340339661 particle/accuracy_1_of_5‚1 epoch‚48 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6693097352981567 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03611055016517639 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5683405995368958 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6378304362297058 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7274854779243469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2220502644777298 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360731482505798 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_3_of_3‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09647992998361588 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6308538317680359 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6919464468955994 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9452828764915466 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2131815254688263 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23621991276741028 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6378304362297058 _step‚1616 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09896577894687653 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31536394357681274 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28279343247413635 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7091246843338013 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7243047952651978 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04078979045152664 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1661490648984909 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.34687697887420654 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7870433330535889 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 *jet/accuracy_1_of_3‚0.9266890287399292 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28309330344200134 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27855268120765686 7validation_average_jet_accuracy‚0.37964314222335815 % _timestamp‚1.7843571313887463e+09 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13543260097503662 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7852444648742676 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.768031895160675 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23797576129436493 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28309330344200134 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08790280669927597 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_3_of_3‚0.03792091831564903 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2413233071565628 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051215432584285736 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23109839856624603 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8287604451179504 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5764477849006653 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397027969360352 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925015687942505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6524272561073303 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 particle/accuracy_5_of_5‚1 trainer/global_step‚19599 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6551302075386047 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *jet/accuracy_1_of_4‚0.9806751012802124 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27468886971473694 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2830818295478821 *jet/accuracy_2_of_2‚0.1260158121585846 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6537662148475647 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7750176191329956 /particle/accuracy_3_of_4‚0.9968254566192627 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6122134327888489 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7272027134895325 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1Ð ËìØYžð;‚(:“ •Âìҧݱ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚274.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.900400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#WÚF.ñ;‚(:£ •ÂìÒü«¶–( proc.memory.availableMB‚ 363.40234375 disk.sda1.in‚ 8.759765625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 51.9111328125" proc.memory.rssMB‚ 2812.0390625 network.sent‚ 3.1490293e+07$ disk./.usageGB‚585.6430435180664 disk.sda.in‚541.0478515625 network.recv‚ 4.452846e+06' disk.nvme0c0n1.out‚35416.2529296875" disk.dm-0.out‚35416.2529296875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 233.44140625 disk.dm-0.in‚28916.26171875 cpu‚26.16255638696753 disk.sda1.out‚0 disk.dm-1.out‚ 1568.15234375 disk.nvme0n1.in‚ 28930.296875 disk.rbd0.in‚ 0.16015625) disk./.usagePercent‚39.30299603774067" disk.nvme0c0n1.in‚ 28930.296875% disk.nvme0n1.out‚35416.2529296875$ memory_percent‚97.78196811676025( proc.memory.percent‚17.1633243560791 disk.drbd2165.out‚0 disk.sda3.out‚ 1568.15234375 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1738.6875 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625{ˆOQœò;‚(:‘ ¤ÂìÒך±“ gpu.0.gpu‚18.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßá K2ó;‚(:§ ¤ÂìÒÀºõ•' disk.nvme0c0n1.out‚35424.5439453125% disk.nvme0n1.out‚35424.5439453125 cpu‚26.162799164957075 network.sent‚ 3.1547608e+07 disk.sda3.in‚ 417.107421875) disk./.usagePercent‚39.30300422998177 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 28930.296875 disk.dm-1.in‚ 233.44140625 proc.cpu.threads‚17 disk.dm-0.in‚28916.26171875 disk.sda3.out‚ 1568.66015625 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.dm-1.out‚ 1568.66015625$ disk./.usageGB‚585.6431655883789 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 51.9111328125 disk.rbd0.out‚0( proc.memory.availableMB‚ 363.32421875( proc.memory.percent‚17.1633243560791 network.recv‚ 4.457021e+06" disk.dm-0.out‚35424.5439453125 disk.sda.out‚ 1739.1953125 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.78244495391846" proc.memory.rssMB‚ 2812.0390625" disk.nvme0c0n1.in‚ 28930.296875 disk.drbd2165.in‚ 2.25390625ýFQ[ô;‚(:’ ³ÂìÒëùÕ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é.œ5õ;‚(:ª ³Âì҆ზ( proc.memory.availableMB‚ 363.32421875 network.recv‚ 4.46049e+06( disk.nvme0c0n1.out‚35429.28759765625" proc.memory.rssMB‚ 2812.0390625" disk.nvme0c0n1.in‚ 28930.296875& disk.nvme0n1.out‚35429.28759765625 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 disk.sda.in‚541.0478515625 cpu‚26.162935702288546$ disk./.usageGB‚585.6430892944336 disk.dm-1.in‚ 233.44140625 disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1569.01953125 proc.cpu.threads‚17 disk.drbd2165.out‚0* disk./.usagePercent‚39.302999109831084 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 28930.296875( proc.memory.percent‚17.1633243560791 network.sent‚ 3.1553344e+07 disk.drbd2165.in‚ 2.25390625# disk.dm-0.out‚35429.28759765625 disk.rbd0.in‚ 0.16015625 disk.dm-1.out‚ 1569.01953125 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 52.0361328125 disk.dm-0.in‚28916.26171875 disk.sda.out‚ 1739.5546875$ memory_percent‚97.78244495391846ô¨jÀ4ö;/ _wandbruntime‚16830  _runtime‚16830ªceÌ÷;Æ % _timestamp‚1.7843571715387852e+09 _runtime‚16830.232393234 _step‚1617 ' lr-AdamW/pg1‚3.5526141884104255e-07 ' lr-AdamW/pg2‚3.5526141884104255e-07 trainer/global_step‚19649(9Ñø;Ë ' lr-AdamW/pg2‚3.5526141884104255e-07 trainer/global_step‚19649 % _timestamp‚1.7843571715387852e+09 _runtime‚16830.232393234 _step‚1617 ' lr-AdamW/pg1‚3.5526141884104255e-07Ñ û<4ù;/ _wandbruntime‚16830  _runtime‚16830š¤õ(ú;P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657778758232064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20761136531727 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20793191638528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42249992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20761477081617 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42249727 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34418608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678796669161984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3008663672379904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337937542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155885294 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2987870839119872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20792850299392 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2972494466551808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21018001604096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2972494483591168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155885294 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34418611 _runtime‚16830.233459207 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155885296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76668603 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843571715396967e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3008664030758400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20793191638528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚19649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34418611 _runtime‚16830.233459207 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155885296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76668603 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843571715396967e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3008664030758400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20793191638528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42249992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20761477081617 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42249727 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34418608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678796669161984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3008663672379904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337937542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155885294 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2987870839119872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20792850299392 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2972494466551808Ò NÊ$h4ü;/ _wandbruntime‚16830  _runtime‚16830Û z'ý;‰N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34420353 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3008812284422144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42251740 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337944247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182051321 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20793862926336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2988018063110144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155892926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2993403482398491 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16830.848296853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2972640977223680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1619 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337954240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2988018063110144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76672093 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155892926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678831129256960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20794204272640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155892928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76672353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182061312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21018942266368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3008811926036480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182051321 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21019026641920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2988018080149504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337944247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678831232244736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657812186990592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20762147585449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3008812284422144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34420356 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16830.848296853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2972640977223680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1619 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337954240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2988018063110144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76672093 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155892926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678831129256960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20794204272640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155892928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76672353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182061312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21018942266368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3008811926036480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182051321 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21019026641920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2988018080149504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337944247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678831232244736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657812186990592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20762147585449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3008812284422144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34420356 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34420353 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3008812284422144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42251740 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337944247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182051321 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20793862926336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304Ó ºØí4ÿ;/ _wandbruntime‚16830  _runtime‚16830"EŸ›€<Ž 0loss/FRt1/assignment_loss‚1.4870678948238492 0loss/FRt2/assignment_loss‚0.6309564597757438 3loss/SRqqt2/assignment_loss‚0.42747393324636745 4loss/FBt1/assignment_loss‚2.0520762201873107e-06 1loss/SRqqt2/detection_loss‚0.5697447389152578 epoch‚49 1loss/SRqqt1/detection_loss‚0.8776468295073078 /loss/FBt2/detection_loss‚0.3314489197636082 &loss/total_loss‚6.7779683073948105 _runtime‚16830.850503058 _step‚1620 /loss/FRt1/detection_loss‚0.5214127128481872 /loss/FRt2/detection_loss‚1.2219417674349402 trainer/global_step‚19649 3loss/SRqqt1/assignment_loss‚0.41127875108209283 3loss/FBt2/assignment_loss‚0.0005727791489304837 0loss/FBt1/detection_loss‚0.29842146877230596 % _timestamp‚1.7843571721572058e+09¾ÅR<“ _runtime‚16830.850503058 _step‚1620 /loss/FRt1/detection_loss‚0.5214127128481872 /loss/FRt2/detection_loss‚1.2219417674349402 trainer/global_step‚19649 3loss/SRqqt1/assignment_loss‚0.41127875108209283 3loss/FBt2/assignment_loss‚0.0005727791489304837 0loss/FBt1/detection_loss‚0.29842146877230596 % _timestamp‚1.7843571721572058e+09 0loss/FRt1/assignment_loss‚1.4870678948238492 0loss/FRt2/assignment_loss‚0.6309564597757438 3loss/SRqqt2/assignment_loss‚0.42747393324636745 4loss/FBt1/assignment_loss‚2.0520762201873107e-06 1loss/SRqqt2/detection_loss‚0.5697447389152578 epoch‚49 1loss/SRqqt1/detection_loss‚0.8776468295073078 /loss/FBt2/detection_loss‚0.3314489197636082 &loss/total_loss‚6.7779683073948105Ô † ï‘‚<‚(:† ÂÂìÒ‚Ûâ“ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¯4Œ.ƒ<‚(:£ ÂÂìңߚ–# disk.nvme0c0n1.in‚ 29056.3984375# disk.dm-0.out‚35435.56689453125 network.sent‚ 3.1557854e+07 disk.dm-0.in‚29042.36328125 disk.sda.in‚541.0478515625$ proc.memory.availableMB‚363.0625 disk.sda1.in‚ 8.759765625 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 29056.3984375$ memory_percent‚97.78404235839844" proc.memory.rssMB‚ 2812.0390625) disk./.usagePercent‚39.30298784549958( proc.memory.percent‚17.1633243560791 cpu‚26.1631606694039 disk.sda2.in‚ 82.544921875 disk.sda3.in‚ 417.107421875 network.recv‚ 4.464012e+06( disk.nvme0c0n1.out‚35435.56689453125 disk.sda.out‚ 1739.80078125 disk.rbd1.out‚ 52.0361328125 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.sda3.out‚ 1569.265625$ disk./.usageGB‚585.6429214477539 disk.drbd2165.in‚ 2.25390625& disk.nvme0n1.out‚35435.56689453125 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1569.265625 disk.drbd2165.out‚0õÆh“„<‚(:ˆ ÑÂìÒà¢ò“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ¦Ö1…<‚(:¦ ÑÂìÒû‚€–! disk.nvme0n1.in‚ 29056.3984375 disk.drbd2165.in‚ 2.25390625 disk.sda.out‚ 1740.14453125$ memory_percent‚97.78404235839844 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625 cpu‚26.163282891165416* disk./.usagePercent‚39.301947430880425 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625# disk.dm-0.out‚35439.80029296875$ proc.memory.availableMB‚363.0625 network.recv‚ 4.468546e+06 disk.dm-1.out‚ 1569.609375( disk.nvme0c0n1.out‚35439.80029296875( proc.memory.percent‚17.1633243560791$ disk./.usageGB‚585.6274185180664 disk.rbd1.out‚ 52.0986328125 disk.rbd0.in‚ 0.16015625# disk.nvme0c0n1.in‚ 29056.3984375" proc.memory.rssMB‚ 2812.0390625 network.sent‚ 3.1620947e+07 proc.cpu.threads‚17 disk.sda3.out‚ 1569.609375 disk.rbd0.out‚0& disk.nvme0n1.out‚35439.80029296875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚29042.36328125 disk.sda2.in‚ 82.544921875Dº»4†</ _wandbruntime‚16870  _runtime‚16870H”¤â̇<Æ _step‚1621 ' lr-AdamW/pg1‚2.6127455525740803e-07 ' lr-AdamW/pg2‚2.6127455525740803e-07 trainer/global_step‚19699 % _timestamp‚1.7843572120007012e+09 _runtime‚16870.696637464Ëã¢Ñˆ<Ë _runtime‚16870.696637464 _step‚1621 ' lr-AdamW/pg1‚2.6127455525740803e-07 ' lr-AdamW/pg2‚2.6127455525740803e-07 trainer/global_step‚19699 % _timestamp‚1.7843572120007012e+09Õ £éwG4‰</ _wandbruntime‚16870  _runtime‚16870Áíï‹Š<€P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680519454346752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21069346765312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784357212004822e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338763615 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3016076419363328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21069258188288 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182506698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34505861 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2995232873562112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16870.698411623 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156266896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2979820000145408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182496721 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.a¤Õzq{llocated‚20843887140352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3000631753164043 _step‚1622 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3016076777741824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2995232890601472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182496721 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42353095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20811753018635 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338773594 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3000632110753293 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3016076419363328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20843545801216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76859221 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338763615 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3016076419363328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21069258188288 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182506698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34505861 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2995232873562112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16870.698411623 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156266896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2979820000145408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182496721 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20843887140352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3000631753164043 _step‚1622 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3016076777741824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2995232890601472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182496721 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42353095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20811753018635 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338773594 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3000632110753293 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3016076419363328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20843545801216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76859221 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680553919602688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3016224673019904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3000779632478999 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2979966510817280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20844899774464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76862690 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20844899774464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2995380131631104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338790292 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3016225031405568 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2995380114591744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚680554016299008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182505773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156274528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20844558428160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42355348 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20812764072357 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3016225031405568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76862954 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20844558428160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156274528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42355087 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659483624916992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7843572126213608e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21070372769792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚19699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182515764 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156274526 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182505773 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338790292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20813104622359 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680553919602688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3016224673019904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3000779632478999 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2979966510817280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20844899774464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76862690 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20844899774464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2995380131631104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338790292 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3016225031405568 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2995380114591744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.alloïµûócated‚680554016299008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182505773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156274528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20844558428160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42355348 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20812764072357 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3016225031405568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76862954 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20844558428160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156274528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42355087 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659483624916992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7843572126213608e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21070372769792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚19699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182515764 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156274526 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182505773 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976× y¿~4</ _wandbruntime‚16871  _runtime‚16871&˜Ñ< %loss/total_loss‚6.839533877068273 epoch‚49 % _timestamp‚1.7843572126235764e+09 _runtime‚16871.316887998 0loss/FBt1/detection_loss‚0.32590493020185163 0loss/FRt1/assignment_loss‚1.3749154001917303 3loss/SRqqt2/assignment_loss‚0.44972175928197783 4loss/FBt1/assignment_loss‚4.0024793566299895e-06 /loss/FRt1/detection_loss‚0.5242910011696691 /loss/FRt2/detection_loss‚1.1041486751104566 _step‚1624 0loss/FRt2/assignment_loss‚0.8249386217006865 2loss/SRqqt1/assignment_loss‚0.5500658577290685 1loss/SRqqt1/detection_loss‚0.8317692011164931 1loss/SRqqt2/detection_loss‚0.5548115565354196 /loss/FBt2/detection_loss‚0.2989367151765054 trainer/global_step‚19699 4loss/FBt2/assignment_loss‚2.6156375058229616e-05Èɤ ‘<’ /loss/FRt2/detection_loss‚1.1041486751104566 _step‚1624 0loss/FRt2/assignment_loss‚0.8249386217006865 2loss/SRqqt1/assignment_loss‚0.5500658577290685 1loss/SRqqt1/detection_loss‚0.8317692011164931 1loss/SRqqt2/detection_loss‚0.5548115565354196 /loss/FBt2/detection_loss‚0.2989367151765054 trainer/global_step‚19699 4loss/FBt2/assignment_loss‚2.6156375058229616e-05 %loss/total_loss‚6.839533877068273 epoch‚49 % _timestamp‚1.7843572126235764e+09 _runtime‚16871.316887998 0loss/FBt1/detection_loss‚0.32590493020185163 0loss/FRt1/assignment_loss‚1.3749154001917303 3loss/SRqqt2/assignment_loss‚0.44972175928197783 4loss/FBt1/assignment_loss‚4.0024793566299895e-06 /loss/FRt1/detection_loss‚0.5242910011696691Ø w®l‘’<‚(:† àÂìҡ gpu.0.gpu‚7.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ¹C¦0“<‚(:¥ àÂìÒ¸—·–& proc.memory.availableMB‚ 363.078125 disk.sda2.in‚ 82.544921875 disk.sda.in‚541.0478515625" proc.memory.rssMB‚ 2812.0390625 disk.sda3.out‚ 1570.0703125 disk.rbd1.in‚ 2.1796875" disk.dm-0.out‚35444.8603515625 disk.sda.out‚ 1740.60546875 disk.rbd0.out‚0 network.sent‚ 3.1626557e+07 disk.rbd0.in‚ 0.16015625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 29056.3984375( proc.memory.percent‚17.1633243560791 proc.cpu.threads‚17 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 29056.3984375 disk.sda3.in‚ 417.107421875 disk.dm-1.out‚ 1570.0703125 disk.drbd2165.in‚ 2.25390625# memory_percent‚97.7839469909668 network.recv‚ 4.471882e+06) disk./.usagePercent‚39.30299680576327$ disk./.usageGB‚585.6430549621582% disk.nvme0n1.out‚35444.8603515625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0 disk.sda1.in‚ 8.759765625 disk.rbd1.out‚ 52.0986328125 cpu‚26.163521178086473' disk.nvme0c0n1.out‚35444.8603515625 disk.dm-0.in‚29042.36328125·çpÅž”<‚(:“ ïÂìÒø÷•“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àbµ¥•<‚(:’ ïÂìÒÝ펕$ disk./.usageGB‚585.6432762145996 disk.sda.out‚ 1741.03515625 disk.drbd2165.in‚ 2.25390625& proc.memory.availableMB‚ 363.140625 cpu‚26.1635281854179 disk.sda.in‚541.0478515625 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625# disk.nvme0c0n1.in‚ 29056.3984375% disk.nvme0c0n1.out‚35448.22265625 disk.dm-0.in‚29042.36328125 disk.dm-0.out‚35448.22265625 disk.sda2.in‚ 82.544921875# disk.nvme0n1.out‚35448.22265625$ memory_percent‚97.78356552124023 disk.sda2.out‚ 170.53515625 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2812.0390625( proc.memory.percent‚17.1633243560791! disk.nvme0n1.in‚ 29056.3984375 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625 network.sent‚ 3.1689763e+07 network.recv‚ 4.476101e+06) disk./.usagePercent‚39.30301165420026 disk.sda3.out‚1570.5 disk.rbd1.out‚ 52.1611328125 disk.rbd0.out‚0 disk.dm-1.out‚1570.5 proc.cpu.threads‚17d¦U=ž–<‚(:“ þÂìÒëШ“ gpu.0.gpu‚46.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.386599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©”G…2—<‚(:§ þÂìÒô×í• disk.dm-1.out‚ 1570.81640625 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1570.81640625' proc.memory.availableMB‚ 363.1484375) disk./.usagePercent‚39.30301037416259 disk.sda.out‚ 1741.3515625% disk.nvme0n1.out‚35452.4853515625 network.sent‚ 3.1694813e+07 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.dm-0.in‚ 29156.65625 proc.cpu.threads‚17 disk.drbd2165.in‚ 2.25390625 disk.dm-1.in‚ 233.44140625' disk.nvme0c0n1.out‚35452.4853515625 disk.sda2.in‚ 82.544921875( proc.memory.percent‚17.1633243560791 disk.sda3.in‚ 417.107421875$ disk.nvme0c0n1.in‚29170.69140625 disk.sda2.out‚ 170.53515625$ memory_percent‚97.78351783752441" proc.memory.rssMB‚ 2812.0390625 cpu‚26.163144714424142$ disk./.usageGB‚585.6432571411133 disk.rbd0.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚35452.4853515625" disk.nvme0n1.in‚29170.69140625 network.recv‚ 4.479493e+06 disk.rbd1.out‚ 52.1611328125 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625.f§&4˜</ _wandbruntime‚16910  _runtime‚16910þŒÅ¡É™<à & lr-AdamW/pg1‚1.816925969264286e-07 & lr-AdamW/pg2‚1.816925969264286e-07 trainer/global_step‚19749 $ _timestamp‚1.784357252265623e+09 _runtime‚16910.959283874 _step‚1625ƒ:®‘Κ<È & lr-AdamW/pg2‚1.816925969264286e-07 trainer/global_step‚19749 $ _timestamp‚1.784357252265623e+09 _runtime‚16910.959283874 _step‚1625 & lr-AdamW/pg1‚1.816925969264286e-07Ù Út•£4›</ _wandbruntime‚16910  _runtime‚16910ÃSÓ(œ<€P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3002594925043712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3002594942083072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3023489524725248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156648494 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682242152078336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156648496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1626 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77049433 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3002594942083072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682242252976128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339599667 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182961150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34593108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661121652697088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21120517993472 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2987145533739008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3023489524725248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3023489166346752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20894582642176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144¦&E. (<…P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3002594942083072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3023489524725248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156648494 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682242152078336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156648496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1626 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77049433 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3002594942083072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682242252976128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339599667 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182961150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34593108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661121652697088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21120517993472 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2987145533739008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3023489524725248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3023489166346752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20894582642176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3002594925043712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20895253929984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3002742183112704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339626344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 " _timestamp‚1.7843572528805e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156656128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77053299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34594856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21121626553856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3008155782559507 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339616351 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156656126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339616351 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682276708009472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20895595276288 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42458446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21121546372608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42458699 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661155062843392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2987292044410880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚16911.574300987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3023637778388992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3002742166073344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2987292061450240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156656128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661155081455616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682276609216000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360—Âç‡' <ŒN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156656128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661155081455616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682276609216000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339626344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20895253929984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3002742183112704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339626344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 " _timestamp‚1.7843572528805e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156656128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77053299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34594856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21121626553856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3008155782559507 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339616351 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156656126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339616351 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682276708009472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20895595276288 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42458446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21121546372608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42458699 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661155062843392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2987292044410880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚16911.574300987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3023637778388992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3002742166073344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2987292061450240Û e§Ý4¡</ _wandbruntime‚16911  _runtime‚16911¦šü¢<ˆ 3loss/FBt1/assignment_loss‚2.476483951951991e-06 1loss/SRqqt1/detection_loss‚0.8683958919182738 %loss/total_loss‚6.953442604841175 4loss/FBt2/assignment_loss‚0.00033414654179618984 /loss/FRt2/detection_loss‚1.1523651818408112 epoch‚49 _runtime‚16911.576433541 /loss/FRt2/assignment_loss‚0.655473772326813 0loss/SRqqt2/detection_loss‚0.588272109323705 /loss/FBt1/detection_loss‚0.4261459173329058 /loss/FBt2/detection_loss‚0.3478361255562726 % _timestamp‚1.7843572528831446e+09 /loss/FRt1/assignment_loss‚1.447830466251088 2loss/SRqqt1/assignment_loss‚0.5029220471156224 3loss/SRqqt2/assignment_loss‚0.41227519066367324 /loss/FRt1/detection_loss‚0.5515892794862628 trainer/global_step‚19749 _step‚1628o£< 3loss/FBt1/assignment_loss‚2.476483951951991e-06 1loss/SRqqt1/detection_loss‚0.8683958919182738 %loss/total_loss‚6.953442604841175 4loss/FBt2/assignment_loss‚0.00033414654179618984 /loss/FRt2/detection_loss‚1.1523651818408112 epoch‚49 _runtime‚16911.576433541 /loss/FRt2/assignment_loss‚0.655473772326813 0loss/SRqqt2/detection_loss‚0.588272109323705 /loss/FBt1/detection_loss‚0.4261459173329058 /loss/FBt2/detection_loss‚0.3478361255562726 % _timestamp‚1.7843572528831446e+09 /loss/FRt1/assignment_loss‚1.447830466251088 2loss/SRqqt1/assignment_loss‚0.5029220471156224 3loss/SRqqt2/assignment_loss‚0.41227519066367324 /loss/FRt1/detection_loss‚0.5515892794862628 trainer/global_step‚19749 _step‚1628Ü ÷`Öœ¤<‚(:‘ ÃìÒµ¢¿“ gpu.0.gpu‚14.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»^(6¥<‚(:« ÃìҨܲ—" disk.nvme0n1.in‚29170.69140625 disk.rbd1.in‚ 2.1796875( proc.memory.percent‚17.1633243560791 cpu‚26.162744645874213 proc.cpu.threads‚17& disk.nvme0n1.out‚35455.76123046875 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.78387546539307 disk.sda2.out‚ 170.53515625 disk.rbd0.in‚ 0.16015625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.in‚ 2.25390625$ disk./.usageGB‚585.6428871154785 disk.dm-0.in‚ 29156.65625 disk.sda2.in‚ 82.544921875# disk.dm-0.out‚35455.76123046875 disk.rbd1.out‚ 52.2236328125 disk.sda1.in‚ 8.759765625( proc.memory.availableMB‚ 363.08984375 disk.sda.in‚541.0478515625 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚29170.69140625" proc.memory.rssMB‚ 2812.0390625 disk.sda.out‚ 1741.6484375 disk.sda3.out‚ 1571.11328125( disk.nvme0c0n1.out‚35455.76123046875 network.sent‚ 3.1699992e+07 network.recv‚ 4.482857e+06) disk./.usagePercent‚39.30298554143177 disk.dm-1.out‚ 1571.11328125¢ßÆIž¦<‚(:“ œÃìÒ­‹¢“ gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œq_”)§<‚(:ž œÃìÒ¶àØ• disk.sda3.in‚ 417.107421875 disk.sda3.out‚ 1571.4375 disk.rbd1.out‚ 52.2236328125 disk.drbd2165.out‚0& disk.nvme0n1.out‚35460.83154296875 proc.cpu.threads‚17 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0( disk.nvme0c0n1.out‚35460.83154296875 disk.sda2.out‚ 170.53515625) disk./.usagePercent‚39.30298554143177 disk.dm-1.in‚ 233.44140625# disk.dm-0.out‚35460.83154296875 disk.sda.out‚ 1741.97265625" disk.nvme0n1.in‚29170.69140625( proc.memory.percent‚17.1633243560791 cpu‚26.16237040156795 network.recv‚ 4.487131e+06 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875% proc.memory.availableMB‚ 362.84375 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625" proc.memory.rssMB‚ 2812.0390625$ disk./.usageGB‚585.6428871154785 disk.rbd1.in‚ 2.1796875 disk.rbd0.in‚ 0.16015625$ disk.nvme0c0n1.in‚29170.69140625 disk.dm-0.in‚ 29156.65625# memory_percent‚97.7853775024414 network.sent‚ 3.176306e+07 disk.dm-1.out‚ 1571.4375ûh “¨<‚(:ˆ «Ãìһꬓ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò'1©<‚(:¦ «ÃìÒ¦Þß• network.recv‚ 4.490593e+06 disk.sda.in‚541.0478515625 disk.sda2.out‚ 170.53515625$ disk./.usageGB‚585.6429634094238$ disk.nvme0c0n1.in‚29170.69140625 disk.sda3.in‚ 417.107421875 disk.rbd1.in‚ 2.1796875" proc.memory.rssMB‚ 2812.0390625 disk.rbd1.out‚ 52.2861328125( proc.memory.availableMB‚ 362.52734375 disk.dm-1.out‚ 1571.59375 disk.sda2.in‚ 82.544921875" disk.nvme0n1.in‚29170.69140625 cpu‚26.16193233526938 disk.rbd0.out‚0 disk.dm-0.in‚ 29156.65625& disk.nvme0n1.out‚35463.62841796875* disk./.usagePercent‚39.302990661582456 disk.drbd2165.in‚ 2.25390625 proc.cpu.threads‚17$ memory_percent‚97.78730869293213 network.sent‚ 3.1768122e+07 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0 disk.sda.out‚ 1742.12890625 disk.rbd0.in‚ 0.16015625( disk.nvme0c0n1.out‚35463.62841796875# disk.dm-0.out‚35463.62841796875 disk.sda3.out‚ 1571.59375 disk.sda1.out‚0( proc.memory.percent‚17.1633243560791H`§‡4ª</ _wandbruntime‚16951  _runtime‚16951}iÓÌ«<Æ ' lr-AdamW/pg1‚1.1652065524857546e-07 ' lr-AdamW/pg2‚1.1652065524857546e-07 trainer/global_step‚19799 % _timestamp‚1.7843572923351762e+09 _runtime‚16951.028778751 _step‚1629AéЉÑ¬<Ë ' lr-AdamW/pg1‚1.1652065524857546e-07 ' lr-AdamW/pg2‚1.1652065524857546e-07 trainer/global_step‚19799 % _timestamp‚1.7843572923351762e+09 _runtime‚16951.028778751 _step‚1629Ý r°Œœ4­</ _wandbruntime‚16951  _runtime‚16951FŠë´K®<€P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77240076 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340435719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚662793090623488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994471067332608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157030096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20945278144000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21171864074752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2994471084371968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3030901913330176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009956976525312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157030096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42559460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3015384410914309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183415602 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20912985992451 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚19799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21171946360320 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large²‘I» _pool.allocated‚34680361 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42559715 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3030902271708672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20913326542341 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3009956993564672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77239818 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚683965036983808 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157030094 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16951.03000741 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20945278144000 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3030902271708672 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16951.03000741 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20945278144000 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3030902271708672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994471067332608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157030096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20945278144000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21171864074752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2994471084371968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3030901913330176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009956976525312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Þ :4°</ _wandbruntime‚16951  _runtime‚16951ˆ ï '±<‹N QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77243628 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34682103 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20914337596175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3031050525372416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3015531932640015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34682106 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1631 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183414677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42561525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183414677 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21172975764480 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3010104217554944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚19799 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183424668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157037726 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183424668 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20913997046173 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3031050166986752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3010104234594304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20946290778112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3015531575050653 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157037726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340462396 % _timestamp‚1.7843572929501853e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157037728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21172893486080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3010104234594304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340452403 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3031050525372416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3015531932640015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34682106 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚1631 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183414677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42561525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183414677 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21172975764480 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3010104217554944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚19799 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183424668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157037726 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183424668 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20913997046173 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3031050166986752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3010104234594304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20946290778112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3015531575050653 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157037726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340462396 % _timestamp‚1.7843572929501853e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157037728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21172893486080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3010104234594304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340452403 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77243628 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77ß ý/¿4³</ _wandbruntime‚16951  _runtime‚16951x¥l ´<† 0loss/FRt2/assignment_loss‚0.7805693225896516 2loss/SRqqt1/assignment_loss‚0.3633820007699638 2loss/SRqqt2/assignment_loss‚0.4445997113323897 0loss/SRqqt2/detection_loss‚0.565388451296154 $ _timestamp‚1.784357292953538e+09 %loss/total_loss‚6.772560578509626 epoch‚49 0loss/FRt1/assignment_loss‚1.4292094747097674 2loss/FBt1/assignment_loss‚0.030639320733255277 3loss/FBt2/assignment_loss‚0.0024282009798159737 /loss/FRt1/detection_loss‚0.5410690917693064 /loss/FRt2/detection_loss‚1.1208317703785038 /loss/FBt1/detection_loss‚0.3467143349252035 /loss/FBt2/detection_loss‚0.2939990667702538 trainer/global_step‚19799 _runtime‚16951.646823412 _step‚1632 1loss/SRqqt1/detection_loss‚0.8537298322553611:Vjòµ<‹ 2loss/FBt1/assignment_loss‚0.030639320733255277 3loss/FBt2/assignment_loss‚0.0024282009798159737 /loss/FRt1/detection_loss‚0.5410690917693064 /loss/FRt2/detection_loss‚1.1208317703785038 /loss/FBt1/detection_loss‚0.3467143349252035 /loss/FBt2/detection_loss‚0.2939990667702538 trainer/global_step‚19799 _runtime‚16951.646823412 _step‚1632 1loss/SRqqt1/detection_loss‚0.8537298322553611 0loss/FRt2/assignment_loss‚0.7805693225896516 2loss/SRqqt1/assignment_loss‚0.3633820007699638 2loss/SRqqt2/assignment_loss‚0.4445997113323897 0loss/SRqqt2/detection_loss‚0.565388451296154 $ _timestamp‚1.784357292953538e+09 %loss/total_loss‚6.772560578509626 epoch‚49 0loss/FRt1/assignment_loss‚1.4292094747097674à ®Æ‡A“¶<‚(:ˆ ºÃìÒ‰Åâ• gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚251.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“û¯6·<‚(:« ºÃìÒžÕ—$ disk./.usageGB‚585.6428260803223 disk.sda.in‚541.0478515625! disk.nvme0n1.in‚ 29309.6328125 disk.rbd1.in‚ 2.1796875 disk.sda1.out‚0 disk.drbd2165.in‚ 2.25390625$ memory_percent‚97.78463840484619 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 29309.6328125# disk.dm-0.out‚35469.91064453125 disk.sda2.out‚ 170.53515625 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 network.recv‚ 4.493061e+06 disk.sda2.in‚ 82.544921875& disk.nvme0n1.out‚35469.91064453125 disk.dm-0.in‚29295.59765625 cpu‚26.161547801784263) disk./.usagePercent‚39.30298144531122 disk.dm-1.out‚ 1571.98046875 network.sent‚ 3.1772345e+07 disk.rbd0.in‚ 0.16015625 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625( proc.memory.availableMB‚ 362.96484375( proc.memory.percent‚17.1633243560791 proc.cpu.threads‚17 disk.sda.out‚ 1742.515625 disk.sda3.out‚ 1571.98046875 disk.rbd1.out‚ 52.2861328125( disk.nvme0c0n1.out‚35469.91064453125" proc.memory.rssMB‚ 2812.0390625c]ä “¸<‚(:ˆ ÉÃìÒ„š´“ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¾Ÿ,¹<‚(:¡ ÉÃìÒû³ò•$ memory_percent‚97.78428077697754 network.recv‚ 4.497558e+06 disk.sda1.in‚ 8.759765625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.out‚0 disk.dm-0.in‚29295.59765625 disk.rbd1.in‚ 2.1796875 disk.rbd0.out‚0) disk./.usagePercent‚39.30299782979341 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 29309.6328125" proc.memory.rssMB‚ 2812.0390625 cpu‚26.161161154614746 disk.sda.in‚541.0478515625 disk.sda3.out‚ 1572.19140625 disk.sda2.in‚ 82.544921875 disk.rbd1.out‚ 52.4111328125 network.sent‚ 3.183492e+07 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 29309.6328125% disk.nvme0c0n1.out‚35472.36328125 disk.dm-0.out‚35472.36328125 proc.cpu.threads‚17 disk.dm-1.out‚ 1572.19140625# disk.nvme0n1.out‚35472.36328125 disk.sda3.in‚ 417.107421875 disk.rbd0.in‚ 0.16015625$ disk./.usageGB‚585.6430702209473 disk.sda1.out‚0 disk.sda.out‚ 1742.7265625' proc.memory.availableMB‚ 363.0234375( proc.memory.percent‚17.1633243560791·(Ù34º</ _wandbruntime‚16990  _runtime‚16990ÛpØ_Ê»<Ä _runtime‚16990.867311769 _step‚1633 & lr-AdamW/pg1‚6.576291609586963e-08 & lr-AdamW/pg2‚6.576291609586963e-08 trainer/global_step‚19849 % _timestamp‚1.7843573321737347e+09¾ÅHϼ<É % _timestamp‚1.7843573321737347e+09 _runtime‚16990.867311769 _step‚1633 & lr-AdamW/pg1‚6.576291609586963e-08 & lr-AdamW/pg2‚6.576291609586963e-08 trainer/global_step‚19849á øò(4½</ _wandbruntime‚16990  _runtime‚16990°î(¾<P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚664464528549888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3017319045046272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21223275933696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20995632306688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19849 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3017319045046272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20995973645824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3017319028006912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3022760560994817 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3001796600926208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77430422 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3038315018692096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20963602479359 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183870054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3038315018692096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685687804483584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77430160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685687701488640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341281750 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157411696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664464509937664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843573321748257e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157411696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157411694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3038314660313600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3038314660313600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21223191550976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3022760203405567 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341271771 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21223275933696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20995632306688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19849 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3017319045046272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20995973645824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3017319028006912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3022760560994817 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3001796600926208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77430422 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3038315018692096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20963602479359 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183870054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3038315018692096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685687804483584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77430160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685687701488640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341281750 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157411696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664464509937664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843573321748257e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157411696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157411694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3038314660313600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3038314660313600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21223191550976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3022760203405567 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341271771 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77433951 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3038463272355840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3017466286075904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21224200793600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3022907725131161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341288455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42664860 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341298448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183869129 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183879120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157419328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34769356 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚664497938696192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20964954083083 _runtime‚16991.481484401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20996644933632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3038462913970176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157419328 HBDeviceStatsMonitor.on_train_batch_end/oáRèÊå versize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341288455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20964613533081 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20996986279936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 $ _timestamp‚1.784357332787685e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3022908082720523 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183869129 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚664497957308416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42664598 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3001943111598080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157419326 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0=²9'Â<N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42664598 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3001943111598080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157419326 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20996986279936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3001943128637440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157419326 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 158196736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77433951 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3038463272355840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3017466286075904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21224200793600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3022907725131161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341288455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42664860 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341298448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183869129 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183879120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157419328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34769356 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚664497938696192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20964954083083 _runtime‚16991.481484401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20996644933632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3038462913970176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157419328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341288455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20964613533081 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20996986279936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 $ _timestamp‚1.784357332787685e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3022908082720523 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183869129 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚664497957308416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚1635ã î´Ÿ4Ã</ _wandbruntime‚16991  _runtime‚16991HnwÄ<ˆ 0loss/FBt2/detection_loss‚0.42650669158884297 trainer/global_step‚19849 2loss/SRqqt1/assignment_loss‚0.4372718305348178 2loss/FBt2/assignment_loss‚0.012968717447385222 epoch‚49 _step‚1636 0loss/FRt1/assignment_loss‚1.4517009611059875 0loss/FRt2/assignment_loss‚0.6885909342378486 3loss/FBt1/assignment_loss‚5.093707743279645e-05 /loss/FRt1/detection_loss‚0.5523094620482966 /loss/FRt2/detection_loss‚1.1822386935033031 1loss/SRqqt1/detection_loss‚0.8475214921344495 /loss/FBt1/detection_loss‚0.5290317078556415 _runtime‚16991.485061674 1loss/SRqqt2/assignment_loss‚0.430954484994227 1loss/SRqqt2/detection_loss‚0.5671938211533694 %loss/total_loss‚7.126339733681602 % _timestamp‚1.7843573327917385e+09&–¤Å< %loss/total_loss‚7.126339733681602 % _timestamp‚1.7843573327917385e+09 0loss/FBt2/detection_loss‚0.42650669158884297 trainer/global_step‚19849 2loss/SRqqt1/assignment_loss‚0.4372718305348178 2loss/FBt2/assignment_loss‚0.012968717447385222 epoch‚49 _step‚1636 0loss/FRt1/assignment_loss‚1.4517009611059875 0loss/FRt2/assignment_loss‚0.6885909342378486 3loss/FBt1/assignment_loss‚5.093707743279645e-05 /loss/FRt1/detection_loss‚0.5523094620482966 /loss/FRt2/detection_loss‚1.1822386935033031 1loss/SRqqt1/detection_loss‚0.8475214921344495 /loss/FBt1/detection_loss‚0.5290317078556415 _runtime‚16991.485061674 1loss/SRqqt2/assignment_loss‚0.430954484994227 1loss/SRqqt2/detection_loss‚0.5671938211533694ä ZÆГÆ<‚(:ˆ ØÃìÒþ¿Ç“ gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚237.057' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.4114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¾­Ð1Ç<‚(:¦ ØÃì҆Ȋ˜ disk.sda3.out‚ 1572.41796875 disk.rbd1.in‚ 2.1796875 disk.dm-1.out‚ 1572.41796875$ proc.memory.availableMB‚362.6875! disk.nvme0n1.in‚ 29309.6328125( proc.memory.percent‚17.1633243560791 network.sent‚ 3.18405e+07 disk.sda3.in‚ 417.107421875 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625# disk.dm-0.out‚35477.67236328125 proc.cpu.threads‚17& disk.nvme0n1.out‚35477.67236328125 disk.drbd2165.out‚0 disk.dm-0.in‚29295.59765625 disk.sda2.out‚ 170.53515625 disk.drbd2165.in‚ 2.25390625# disk.nvme0c0n1.in‚ 29309.6328125" proc.memory.rssMB‚ 2812.0390625 network.recv‚ 4.500829e+06* disk./.usagePercent‚39.303025990622174$ disk./.usageGB‚585.6434898376465( disk.nvme0c0n1.out‚35477.67236328125 disk.sda.in‚541.0478515625 disk.sda.out‚ 1742.953125 disk.rbd1.out‚ 52.4111328125 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625$ memory_percent‚97.78633117675781 cpu‚26.160762468318886 disk.sda2.in‚ 82.544921875 disk.rbd0.in‚ 0.16015625ÔC7d“È<‚(:ˆ çÃìÒòŠÖ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àŠÌÈ"É<‚(:— çÃìÒƒ¸– disk.dm-0.in‚29295.59765625 disk.sda3.out‚ 1572.734375$ memory_percent‚97.78711795806885( proc.memory.availableMB‚ 362.55859375 network.recv‚ 4.505086e+06 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0 disk.rbd0.in‚ 0.16015625 disk.rbd0.out‚0 disk.dm-0.out‚ 35481.46875 disk.sda.out‚ 1743.26953125 disk.nvme0n1.out‚ 35481.46875 disk.sda3.in‚ 417.107421875 network.sent‚ 3.1903028e+07 disk.sda.in‚541.0478515625 disk.rbd1.out‚ 52.4736328125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 29309.6328125 disk.sda2.in‚ 82.544921875" proc.memory.rssMB‚ 2812.0390625" disk.nvme0c0n1.out‚ 35481.46875 disk.rbd1.in‚ 2.1796875( proc.memory.percent‚17.1633243560791 cpu‚26.160362108663847) disk./.usagePercent‚39.30198634402563$ disk./.usageGB‚585.6279983520508 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625! disk.nvme0n1.in‚ 29309.6328125 disk.dm-1.in‚ 233.44140625 disk.dm-1.out‚ 1572.734375 disk.drbd2165.out‚0ø¢¥‚“Ê<‚(:ˆ öÃìÒòµ“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚241.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.2714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æû„¥6Ë<‚(:« öÃìÒ§ãÝ• disk.rbd1.in‚ 2.1796875 disk.dm-1.out‚ 1573.0234375 disk.sda3.in‚ 417.107421875 network.sent‚ 3.1908113e+07 disk.rbd1.out‚ 52.4736328125 network.recv‚ 4.508077e+06$ disk./.usageGB‚585.6280784606934 disk.sda1.out‚0 disk.dm-1.in‚ 233.44140625# disk.dm-0.out‚35487.85107421875 disk.sda.in‚541.0478515625" proc.memory.rssMB‚ 2812.0390625 proc.cpu.threads‚17 disk.rbd0.in‚ 0.16015625 disk.dm-0.in‚29342.23828125 disk.sda.out‚ 1743.55859375( proc.memory.availableMB‚ 362.55859375 cpu‚26.159974107114156( disk.nvme0c0n1.out‚35487.85107421875) disk./.usagePercent‚39.30199172018385 disk.rbd0.out‚0 disk.drbd2165.in‚ 2.25390625 disk.sda2.out‚ 170.53515625 disk.sda3.out‚ 1573.0234375( proc.memory.percent‚17.1633243560791 disk.sda2.in‚ 82.544921875& disk.nvme0n1.out‚35487.85107421875 disk.drbd2165.out‚0 disk.sda1.in‚ 8.759765625# disk.nvme0c0n1.in‚ 29356.2734375! disk.nvme0n1.in‚ 29356.2734375$ memory_percent‚97.78711795806885Ê™~K4Ì</ _wandbruntime‚17030  _runtime‚17030V ÅÊÍ<Ä & lr-AdamW/pg1‚2.942263954304994e-08 & lr-AdamW/pg2‚2.942263954304994e-08 trainer/global_step‚19899 % _timestamp‚1.7843573722327538e+09 _runtime‚17030.926305387 _step‚1637atæ'ÏÎ<É & lr-AdamW/pg2‚2.942263954304994e-08 trainer/global_step‚19899 % _timestamp‚1.7843573722327538e+09 _runtime‚17030.926305387 _step‚1637 & lr-AdamW/pg1‚2.942263954304994e-08å >‹LÎ4Ï</ _wandbruntime‚17030  _runtime‚17030cöí’(Ð<P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184314529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21046669147648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666135966476288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21046327808512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚687410497349632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184314529 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚19899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21046327808512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3045727407297024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚687410394354688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42765886 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3045727407297024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77620747 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3030136711075325 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21274530873344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342107823 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3024681096527872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21014218966267 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342117802 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3045727765675520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34854858 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3009122151559168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21046669147648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34854861 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3045727765675520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3030136353486075 _runtime‚17030.927528005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157793294 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3024681079488512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21274446490624 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157793296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157793296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÐJ]Ãð Ñ<†P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157793294 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3024681079488512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21274446490624 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157793296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157793296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184314529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21046669147648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666135966476288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21046327808512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚687410497349632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_baѳTLtch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184314529 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚19899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21046327808512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3045727407297024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚687410394354688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42765886 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3045727407297024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77620747 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3030136711075325 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21274530873344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342107823 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3024681096527872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21014218966267 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342117802 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3045727765675520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34854858 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3009122151559168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21046669147648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34854861 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3045727765675520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3030136353486075 _runtime‚17030.927528005æ M‹€4Ò</ _wandbruntime‚17031  _runtime‚17031£¹_'Ó<ŠN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184323581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3045876019339264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21275556783616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1639 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3030284232801031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34856606 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184333572 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342134500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342124507 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184323581 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3045876019339264 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157800926 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77624313 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚687444952018432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42767949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚17031.541963971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157800926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77624555 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21047681781760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3045875660953600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21047340435456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157800928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3024828320518144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42767710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666169395234816 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157800928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19899 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3030283875211669 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3024828337557504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21047681781760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34856603 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184333572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3030283875211669 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3024828337557504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21047681781760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34856603 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184333572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3045876019339264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21275556783616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1639 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3030284232801031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34856606 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184333572 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342134500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342124507 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184323581 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3045876019339264 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157800926 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77624313 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚687444952018432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42767949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚17031.541963971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157800926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77624555 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21047681781760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3045875660953600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21047340435456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157800928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3024828320518144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42767710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666169395234816 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157800928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19899 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184768981 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21325685037056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21097023310336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19949 _runtime‚17071.572554286 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158174894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77811759 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3053140154280448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚689133070827520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42869914 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3053140154280448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3032043130970112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3032043130970112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3037512503566583 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144CÞ„R (ã<…P _runtime‚17071.572554286 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158174894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77811759 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3053140154280448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚689133070827520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42869914 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3053140154280448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3032043130970112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3032043130970112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3037512503566583 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184778958 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3016447685152768 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158174896 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3032043148009472 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3053140512658944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21325767322624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184768981 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77812025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34942111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚689133171725312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342953854 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42869651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184768981 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21325685037056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21097023310336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1642 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19949ê ´/Óµ4ä</ _wandbruntime‚17072  _runtime‚17072YÉKÿ'å<‹N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184788024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21098377283584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3053288407937024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342970552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1643 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3037660382881539 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚17072.185671921 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34943856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42871923 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3037660025292177 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342960559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42871651 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667840814548992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77815504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚689167523918336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342970552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3032190389039104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667840833161216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3032190371999744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21098035937280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21326709369344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21065846506897 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21326793744896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34943853 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3016594195824640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚19949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158182526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843574134919097e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667840814548992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_spli&ÊC6=%t.all.freed‚77815504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚689167523918336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342970552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3032190389039104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667840833161216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3032190371999744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21098035937280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21326709369344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21065846506897 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21326793744896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34943853 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3016594195824640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚19949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158182526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843574134919097e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3053288407937024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342970552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1643 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3037660382881539 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚17072.185671921 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34943856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42871923 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3037660025292177 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342960559ë @=á04ç</ _wandbruntime‚17072  _runtime‚17072ã¸9è<Œ 0loss/FBt2/detection_loss‚0.34699525349182053 trainer/global_step‚19949 _step‚1644 3loss/SRqqt2/assignment_loss‚0.42961156306108883 /loss/FRt1/detection_loss‚0.5274628447732618 1loss/SRqqt2/detection_loss‚0.5826155881017998 0loss/FBt1/detection_loss‚0.44314400284102623 &loss/total_loss‚6.8764665056330845 % _timestamp‚1.7843574134956145e+09 _runtime‚17072.18905629 0loss/FRt1/assignment_loss‚1.3119297334701148 0loss/FRt2/assignment_loss‚0.7333952077115585 3loss/FBt1/assignment_loss‚0.0007535391796286655 2loss/FBt2/assignment_loss‚0.008572477976336585 /loss/FRt2/detection_loss‚1.2653791489640098 epoch‚49 3loss/SRqqt1/assignment_loss‚0.46733795365832015 1loss/SRqqt1/detection_loss‚0.75926919240411952tPé<‘ 3loss/SRqqt2/assignment_loss‚0.42961156306108883 /loss/FRt1/detection_loss‚0.5274628447732618 1loss/SRqqt2/detection_loss‚0.5826155881017998 0loss/FBt1/detection_loss‚0.44314400284102623 &loss/total_loss‚6.8764665056330845 % _timestamp‚1.7843574134956145e+09 _runtime‚17072.18905629 0loss/FRt1/assignment_loss‚1.3119297334701148 0loss/FRt2/assignment_loss‚0.7333952077115585 3loss/FBt1/assignment_loss‚0.0007535391796286655 2loss/FBt2/assignment_loss‚0.008572477976336585 /loss/FRt2/detection_loss‚1.2653791489640098 epoch‚49 3loss/SRqqt1/assignment_loss‚0.46733795365832015 1loss/SRqqt1/detection_loss‚0.7592691924041195 0loss/FBt2/detection_loss‚0.34699525349182053 trainer/global_step‚19949 _step‚1644ì BÑ¥_“ê<‚(:ˆ ²ÄìÒÓ•Ó“ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÉ€b3ë<‚(:¨ ²ÄìÒ„Ú°—) disk./.usagePercent‚39.30304339913451 disk.drbd2165.in‚ 2.25390625 disk.rbd1.in‚ 2.1796875 disk.rbd1.out‚ 52.5986328125 disk.rbd0.out‚0 disk.sda1.in‚ 8.759765625 proc.cpu.threads‚17 network.sent‚ 3.1986997e+07 disk.dm-1.in‚ 233.44140625 disk.sda.out‚ 1744.90234375 disk.sda2.out‚ 170.53515625" disk.nvme0n1.in‚29465.98828125 disk.sda1.out‚0( proc.memory.percent‚17.1633243560791' disk.nvme0c0n1.out‚35504.8134765625$ disk./.usageGB‚585.6437492370605 disk.drbd2165.out‚0" disk.dm-0.out‚35504.8134765625 disk.rbd0.in‚ 0.16015625$ memory_percent‚97.78711795806885 cpu‚26.160518209831725$ disk.nvme0c0n1.in‚29465.98828125 disk.sda3.in‚ 417.107421875" proc.memory.rssMB‚ 2812.0390625 network.recv‚ 4.522603e+06 disk.sda.in‚541.0478515625 disk.sda2.in‚ 82.544921875( proc.memory.availableMB‚ 362.55859375 disk.dm-1.out‚ 1574.3671875 disk.dm-0.in‚ 29451.953125% disk.nvme0n1.out‚35504.8134765625 disk.sda3.out‚ 1574.3671875€â#’ì<‚(:‡ ÁÄìÒ¯šœ“ gpu.0.gpu‚30.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.4212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cý3í<‚(:¨ ÁÄìÒ‘‰”— disk.rbd1.in‚ 2.1796875& disk.nvme0n1.out‚35508.41357421875 disk.sda3.out‚ 1574.73046875( proc.memory.percent‚17.1633243560791" disk.nvme0c0n1.in‚ 29586.984375" proc.memory.rssMB‚ 2812.0390625 proc.cpu.threads‚17 disk.sda.out‚ 1745.265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚35508.41357421875) disk./.usagePercent‚39.30304928730779# disk.dm-0.out‚35508.41357421875 disk.rbd1.out‚ 52.6611328125 disk.rbd0.in‚ 0.16015625 network.sent‚ 3.2050104e+07 network.recv‚ 4.527063e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 1574.73046875 disk.sda3.in‚ 417.107421875 disk.sda1.in‚ 8.759765625 disk.sda1.out‚0$ disk./.usageGB‚585.6438369750977 disk.dm-0.in‚29572.94921875 disk.sda.in‚541.0478515625 disk.dm-1.in‚ 233.44140625 disk.drbd2165.in‚ 2.25390625( proc.memory.availableMB‚ 362.87109375 cpu‚26.16074327320551$ memory_percent‚97.78521060943604 disk.sda2.in‚ 82.544921875 disk.sda2.out‚ 170.53515625 disk.nvme0n1.in‚ 29586.984375²ãA4î</ _wandbruntime‚17111  _runtime‚17111üî…Éï<à & lr-AdamW/pg1‚2.884352707721945e-12 & lr-AdamW/pg2‚2.884352707721945e-12 trainer/global_step‚19999 $ _timestamp‚1.784357453244749e+09 _runtime‚17111.938286241 _step‚1645IÎÂÎð<È trainer/global_step‚19999 $ _timestamp‚1.784357453244749e+09 _runtime‚17111.938286241 _step‚1645 & lr-AdamW/pg1‚2.884352707721945e-12 & lr-AdamW/pg2‚2.884352707721945e-12í ®± ¨4ñ</ _wandbruntime‚17111  _runtime‚17111ÞôÔ4(ò<P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158556496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158556494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚690855708799488 % _timestamp‚1.7843574532458375e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158556494 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185223433 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35029361 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3039405182451712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343779927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78002191 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158556496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669478842329088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185233410 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3044888653647091 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35029358 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21115451940083 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3039405182451712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78001912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21147718812160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21376982048256 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669478823716864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3060552901263872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚19999 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3044889011236341 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1646 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21148060151296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42972830 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3023773218746368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3060552901263872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343789906 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21376885082624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21115792489973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21148060151296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚170393601™só<†P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35029361 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3039405182451712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343779927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78002191 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158556496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669478842329088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185233410 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 158196736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3044888653647091 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35029358 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21115451940083 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3039405182451712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78001912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21147718812160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21376982048256 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669478823716864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3060552901263872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚19999 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3044889011236341 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1646 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21148060151296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42972830 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3023773218746368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3060552901263872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocÔIa³™ ated‚ 343789906 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21376885082624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21115792489973 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21148060151296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158556496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158556494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚690855708799488 % _timestamp‚1.7843574532458375e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158556494 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185223433 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632î ªS=4ô</ _wandbruntime‚17112  _runtime‚17112Nkx 'õ<‹N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78005922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185242476 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35031103 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158564128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21116803543807 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343806604 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21116462993805 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343796611 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185242476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3045036532962047 trainer/global_step‚19999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21148731439104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3039552440520704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21377914379264 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3045036175372685 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343796611 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35031106 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21149072785408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185232485 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚690890166854656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚17112.553080845 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3060701154920448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78005665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158564128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21116803543807 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343806604 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21116462993805 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343796611 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185242476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3045036532962047 trainer/global_step‚19999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21148731439104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3039552440520704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21377914379264 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3045036175372685 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343796611 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35031106 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21149072785408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185232485 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚690890166854656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚17112.553080845 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3060701154920448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78005665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3631744682788849 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21302740275859833 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6524272561073303 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6572780013084412 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_3‚1 epoch‚49 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37647250294685364 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782057166099548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6547632813453674 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.786576509475708 +particle/sensitivity‚0.5952824354171753 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373721361160278 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2413233071565628 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360731482505798 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6823356747627258 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09896577894687653 (particle/accuracy‚0.8331860899925232 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5520079135894775 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.567487359046936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6916531324386597 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 (loss/val_total_loss‚7.70579820028828 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27454155683517456 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7269585132598877 (jet/accuracy_1_of_3‚0.92719566822052 'particle/f_score‚0.6051508188247681 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 _step‚1649 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_2_of_3‚0.5107715129852295 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14486175775527954 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7337389588356018 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31515276432037354 7 Purity/*F/×)IÚ'Rt1SRqqt1FBt/FRt_purity‚0.5077018737792969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1352797895669937 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24934005737304688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 +jet/accuracy_2_of_2‚0.12611910700798035 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5764477849006653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15190909802913666 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23209044337272644 _runtime‚17130.592882011 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5683405995368958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7136529088020325 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6677929759025574 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7089664340019226 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8287604451179504 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6690131425857544 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.653419017791748 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09796803444623947 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24910306930541992 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 $ _timestamp‚1.784357471898431e+09 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6117817759513855 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.247840017080307 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14928790926933289 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6281405687332153 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_4_of_4‚0.9499987363815308 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23608368635177612 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16614258289337158 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373721361160278 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3380573093891144 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23782171308994293 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7920346856117249 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7848833799362183 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2829364538192749 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7678248882293701 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9806751012802124 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6308538317680359 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7272307276725769 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 trainer/global_step‚19999 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23811590671539307 6validation_average_jet_accuracy‚0.3794363737106323 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2221267968416214 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 +jet/accuracy_3_of_4‚0.04078979045152664 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21237456798553467 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7243047952651978 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24590946733951569 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28297775983810425 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04559728130698204 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28260743618011475 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2780764698982239 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2936764359474182 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03618636354804039 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2958303391933441 *jet/accuracy_1_of_1‚0.3466044068336487 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32948923110961914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6830228567123413 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397027969360352 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051215432584285736 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08790280669927597 *jet/accuracy_1_of_2‚0.7850279808044434 particle/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_3_of_3‚0.03792091831564903 particle/accuracy_2_of_2‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7748068571090698 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.707197904586792 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29344642162323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.638312816619873 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904703974723816 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2829364538192749 *jet/accuracy_2_of_4‚0.7723941802978516+$#Wƒ=€=ýz 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7748068571090698 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.178550586104393 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959112524986267 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8739367723464966 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.707197904586792 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29344642162323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.638312816619873 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6904703974723816 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7623906135559082 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2829364538192749 *jet/accuracy_2_of_4‚0.7723941802978516 +particle/specificity‚0.8982466459274292 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03597252815961838 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8478268980979919 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02456441894173622 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957776069641113 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3631744682788849 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21302740275859833 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024792086333036423 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6524272561073303 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6572780013084412 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_3‚1 epoch‚49 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37647250294685364 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782057166099548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6547632813453674 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.786576509475708 +particle/sensitivity‚0.5952824354171753 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974469542503357 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6373721361160278 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2413233071565628 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360731482505798 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6823356747627258 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09896577894687653 (particle/accuracy‚0.8331860899925232 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5520079135894775 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977415800094604 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.567487359046936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6916531324386597 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 (loss/val_total_loss‚7.70579820028828 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960101246833801 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27454155683517456 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7269585132598877 (jet/accuracy_1_of_3‚0.92719566822052 'particle/f_score‚0.6051508188247681 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 _step‚1649 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_2_of_3‚0.5107715129852295 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998171329498291 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14486175775527954 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_4‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11968269944190979 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7337389588356018 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31515276432037354 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5077018737792969 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8478268980979919 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1352797895669937 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24934005737304688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8739367723464966 +jet/accuracy_2_of_2‚0.12611910700798035 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.998236358165741 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5764477849006653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15190909802913666 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23209044337272644 _runtime‚17130.592882011 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5683405995368958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7136529088020325 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7944472432136536 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6677929759025574 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8211262226104736 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7089664340019226 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8287604451179504 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6690131425857544 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.653419017791748 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09796803444623947 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5041018724441528 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938538670539856 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02492348663508892 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24910306930541992 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 $ _timestamp‚1.784357471898431e+09 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.72584468126297 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6117817759513855 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.247840017080307 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14928790926933289 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6281405687332153 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_4_of_4‚0.9499987363815308 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23608368635177612 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16614258289337158 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6373721361160278 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3380573093891144 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888337254524231 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025106213986873627 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23782171308994293 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7428332567214966 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7920346856117249 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7848833799362183 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2829364538192749 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.032256387174129486 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984288811683655 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7678248882293701 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9806751012802124 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6308538317680359 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7272307276725769 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 * Purity/2FRt0SRqqt1FBt/FRt_purity‚0.625 trainer/global_step‚19999 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23811590671539307 6validation_average_jet_accuracy‚0.3794363737106323 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9951794147491455 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2221267968416214 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7709510326385498 +jet/accuracy_3_of_4‚0.04078979045152664 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949188232421875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8601662516593933 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21237456798553467 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7243047952651978 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9406936168670654 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24590946733951569 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28297775983810425 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04559728130698204 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28260743618011475 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2780764698982239 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2936764359474182 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03618636354804039 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10083333402872086 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2958303391933441 *jet/accuracy_1_of_1‚0.3466044068336487 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32948923110961914 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5412499904632568 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6830228567123413 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7249776124954224 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023889027535915375 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974469542503357 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6397027969360352 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023889027535915375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051215432584285736 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08790280669927597 *jet/accuracy_1_of_2‚0.7850279808044434 particle/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_3_of_3‚0.03792091831564903 particle/accuracy_2_of_2‚1ñ S!DÜR= wclzbk8uj@ ßÄì҈뾼0`Trainer.fit` stopped: `max_epochs=50` reached. ‚ä¢t¡‚= wclzbk8ujŽ àÄìÒ Æüö|Epoch 49/49 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 400/400 0:05:22 • 0:00:00 1.23it/s v_num: bk8uU³¯‡%ƒ= wclzbk8uj àÄìÒˆŒ’÷ ¸ä¦ï4„=/ _wandbruntime‚17132  _runtime‚17132‰ôBñ…=‚(’ì…